[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:31:03 dist-test-slave-dt7p krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:31:03 dist-test-slave-dt7p krb5kdc[395](info): set up 2 sockets
Sep 26 03:31:03 dist-test-slave-dt7p krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 03:31:04.366716 388 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:31:04 dist-test-slave-dt7p krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:31:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:31:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:31:04.561435 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45431
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45431
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:04.956215 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:04.956674 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:04.956843 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:04.964545 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:31:04.964741 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:04.964849 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:04.964979 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:31:04.965070 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:31:04.976390 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:04.978217 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:04.979995 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:04.995435 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:04.995541 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:04.995517 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:04.995951 408 server_base.cc:1034] running on GCE node
I20260926 03:31:04.997027 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:04.998690 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:04.999912 408 hybrid_clock.cc:648] HybridClock initialized: now 1790393464999821 us; error 71 us; skew 500 ppm
I20260926 03:31:05.000303 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:04 dist-test-slave-dt7p krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:05.003896 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:31:05.005766 408 webserver.cc:460] Webserver started at http://127.0.97.62:35635/ using document root <none> and password file <none>
I20260926 03:31:05.006661 408 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:05.006821 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:05.007187 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:05.011005 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "8020d651374048699d4d7cfad5a22bfc"
format_stamp: "Formatted at 2026-09-26 03:31:05 on dist-test-slave-dt7p"
server_key: "e356ae35781fd6e0c73edcaa64d6931b"
server_key_iv: "d0a352ceffb58d8fe9de1ea77bd36596"
server_key_version: "encryptionkey@0"
I20260926 03:31:05.011979 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "8020d651374048699d4d7cfad5a22bfc"
format_stamp: "Formatted at 2026-09-26 03:31:05 on dist-test-slave-dt7p"
server_key: "e356ae35781fd6e0c73edcaa64d6931b"
server_key_iv: "d0a352ceffb58d8fe9de1ea77bd36596"
server_key_version: "encryptionkey@0"
I20260926 03:31:05.019376 408 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260926 03:31:05.023890 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:05.025849 408 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 03:31:05.026100 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "8020d651374048699d4d7cfad5a22bfc"
format_stamp: "Formatted at 2026-09-26 03:31:05 on dist-test-slave-dt7p"
server_key: "e356ae35781fd6e0c73edcaa64d6931b"
server_key_iv: "d0a352ceffb58d8fe9de1ea77bd36596"
server_key_version: "encryptionkey@0"
I20260926 03:31:05.026357 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:05.044698 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:05.050505 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:05.050981 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:05.075053 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45431
I20260926 03:31:05.075076 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45431 every 8 connection(s)
I20260926 03:31:05.077306 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:31:05.082593 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:05.083531 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 408
I20260926 03:31:05.083911 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:31:05.087697 388 external_mini_cluster.cc:1428] Setting key c97c841f5235fccaed14f6804efcb931
I20260926 03:31:05.100207 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8020d651374048699d4d7cfad5a22bfc: Bootstrap starting.
I20260926 03:31:05.106185 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8020d651374048699d4d7cfad5a22bfc: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:05.107896 475 log.cc:824] T 00000000000000000000000000000000 P 8020d651374048699d4d7cfad5a22bfc: Log is configured to *not* fsync() on all Append() calls
I20260926 03:31:05.112190 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8020d651374048699d4d7cfad5a22bfc: No bootstrap required, opened a new log
I20260926 03:31:05.123660 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8020d651374048699d4d7cfad5a22bfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8020d651374048699d4d7cfad5a22bfc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45431 } }
I20260926 03:31:05.124265 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8020d651374048699d4d7cfad5a22bfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:05.124387 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8020d651374048699d4d7cfad5a22bfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8020d651374048699d4d7cfad5a22bfc, State: Initialized, Role: FOLLOWER
I20260926 03:31:05.125315 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8020d651374048699d4d7cfad5a22bfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8020d651374048699d4d7cfad5a22bfc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45431 } }
I20260926 03:31:05.125622 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8020d651374048699d4d7cfad5a22bfc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:31:05.125777 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8020d651374048699d4d7cfad5a22bfc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:31:05.126003 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8020d651374048699d4d7cfad5a22bfc [term 0 FOLLOWER]: Advancing to term 1
Sep 26 03:31:05 dist-test-slave-dt7p krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:31:05.129612 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8020d651374048699d4d7cfad5a22bfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8020d651374048699d4d7cfad5a22bfc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45431 } }
I20260926 03:31:05.130259 475 leader_election.cc:302] T 00000000000000000000000000000000 P 8020d651374048699d4d7cfad5a22bfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8020d651374048699d4d7cfad5a22bfc; no voters:
I20260926 03:31:05.131090 475 leader_election.cc:288] T 00000000000000000000000000000000 P 8020d651374048699d4d7cfad5a22bfc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:31:05.131192 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8020d651374048699d4d7cfad5a22bfc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:31:05.132081 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8020d651374048699d4d7cfad5a22bfc [term 1 LEADER]: Becoming Leader. State: Replica: 8020d651374048699d4d7cfad5a22bfc, State: Running, Role: LEADER
I20260926 03:31:05.133790 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8020d651374048699d4d7cfad5a22bfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8020d651374048699d4d7cfad5a22bfc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45431 } }
I20260926 03:31:05.135994 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8020d651374048699d4d7cfad5a22bfc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:31:05.137130 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8020d651374048699d4d7cfad5a22bfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8020d651374048699d4d7cfad5a22bfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8020d651374048699d4d7cfad5a22bfc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45431 } } }
I20260926 03:31:05.137571 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8020d651374048699d4d7cfad5a22bfc [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:05.138166 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8020d651374048699d4d7cfad5a22bfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8020d651374048699d4d7cfad5a22bfc. Latest consensus state: current_term: 1 leader_uuid: "8020d651374048699d4d7cfad5a22bfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8020d651374048699d4d7cfad5a22bfc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45431 } } }
I20260926 03:31:05.138516 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8020d651374048699d4d7cfad5a22bfc [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:05.141582 485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:31:05.148291 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:05.093646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56038 (local address 127.0.97.62:45431)
0926 03:31:05.094902 (+ 1256us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:05.094942 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:05.096378 (+ 1436us) server_negotiation.cc:407] Connection header received
0926 03:31:05.110944 (+ 14566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:05.110974 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:05.111565 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:05.112019 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:05.116949 (+ 4930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:05.119115 (+ 2166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:05.121969 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:05.122781 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:05.122945 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:05.131003 (+ 8058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:05.131057 (+ 54us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:05.131083 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:05.131151 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:05.136835 (+ 5684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:05.138160 (+ 1325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:05.138171 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:05.138193 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:05.138429 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:05.139075 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:05.139080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:05.139083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:05.139795 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:05.140147 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:05.142904 (+ 2757us) server_negotiation.cc:299] Negotiation successful
0926 03:31:05.143378 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":893,"thread_start_us":522,"threads_started":1}
I20260926 03:31:05.150162 485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:31:05.162885 485 catalog_manager.cc:1347] Generated new cluster ID: 70f981eef3834c74877a8b1ca26436c8
I20260926 03:31:05.163048 485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:31:05.197163 485 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:31:05.198762 485 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:31:05.210006 485 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8020d651374048699d4d7cfad5a22bfc: Generated new TSK 0
I20260926 03:31:05.211184 485 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:31:05.265484 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45431
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:05.676842 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:05.677182 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:05.677353 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:05.684943 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:05.685055 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:05.685264 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:31:05.695590 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45431
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:05.697192 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:05.698800 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:05.714275 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:05.714390 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:05.714377 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:05.718067 501 server_base.cc:1034] running on GCE node
I20260926 03:31:05.718765 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:05.719852 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:05.721091 501 hybrid_clock.cc:648] HybridClock initialized: now 1790393465721043 us; error 43 us; skew 500 ppm
I20260926 03:31:05.721458 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:05 dist-test-slave-dt7p krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393465, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:05.725642 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:31:05.727415 501 webserver.cc:460] Webserver started at http://127.0.97.1:40753/ using document root <none> and password file <none>
I20260926 03:31:05.728237 501 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:05.728338 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:05.728698 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:05.731930 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-0/data/instance:
uuid: "ad0e1f2d87a14961a7f3479cc1d4c696"
format_stamp: "Formatted at 2026-09-26 03:31:05 on dist-test-slave-dt7p"
server_key: "41851c160ff601ac906d3d064095dd9c"
server_key_iv: "da64d7db813e2cb63e8e0314f6152730"
server_key_version: "encryptionkey@0"
I20260926 03:31:05.732808 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ad0e1f2d87a14961a7f3479cc1d4c696"
format_stamp: "Formatted at 2026-09-26 03:31:05 on dist-test-slave-dt7p"
server_key: "41851c160ff601ac906d3d064095dd9c"
server_key_iv: "da64d7db813e2cb63e8e0314f6152730"
server_key_version: "encryptionkey@0"
I20260926 03:31:05.738698 501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260926 03:31:05.742672 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:05.744241 501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:31:05.744423 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-0/wal
uuid: "ad0e1f2d87a14961a7f3479cc1d4c696"
format_stamp: "Formatted at 2026-09-26 03:31:05 on dist-test-slave-dt7p"
server_key: "41851c160ff601ac906d3d064095dd9c"
server_key_iv: "da64d7db813e2cb63e8e0314f6152730"
server_key_version: "encryptionkey@0"
I20260926 03:31:05.744700 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:05.763826 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:05.770159 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:05.770525 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:05.771751 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:31:05.773433 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:31:05.773545 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:05.773710 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:31:05.773787 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:05.813470 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45667
I20260926 03:31:05.813501 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45667 every 8 connection(s)
I20260926 03:31:05.815064 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:31:05.818603 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 501
I20260926 03:31:05.818868 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:31:05.819569 388 external_mini_cluster.cc:1428] Setting key 6baf363c25dc2b86ba47172c6abff7b6
Sep 26 03:31:05 dist-test-slave-dt7p krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393465, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 03:31:05.840171 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:05.818762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57925 (local address 127.0.97.62:45431)
0926 03:31:05.820091 (+ 1329us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:05.820098 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:05.820848 (+ 750us) server_negotiation.cc:407] Connection header received
0926 03:31:05.823026 (+ 2178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:05.823035 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:05.823140 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:05.823276 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:05.825065 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:05.826290 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:05.827986 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:05.828417 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:05.828558 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:05.832872 (+ 4314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:05.832877 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:05.832881 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:05.832925 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:05.835685 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:05.836758 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:05.836762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:05.836764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:05.836907 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:05.837415 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:05.837419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:05.837421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:05.837768 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:05.837886 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:05.839612 (+ 1726us) server_negotiation.cc:299] Negotiation successful
0926 03:31:05.839808 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1183,"thread_start_us":1037,"threads_started":1}
I20260926 03:31:05.841492 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:05.819648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45431 (local address 127.0.97.1:57925)
0926 03:31:05.820668 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:05.820728 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:05.822710 (+ 1982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:05.823520 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:05.823529 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:05.824118 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:05.824878 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:05.824913 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:05.826447 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:05.826451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:05.827834 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:05.827846 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:05.828698 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:05.828709 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:05.828952 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:05.829707 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:05.829735 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:05.832650 (+ 2915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:05.835843 (+ 3193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:05.835853 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:05.835874 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:05.836640 (+ 766us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:05.837024 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:05.837033 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:05.837038 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:05.837312 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:05.837897 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:05.837909 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:05.839157 (+ 1248us) client_negotiation.cc:769] Sending connection context
0926 03:31:05.839508 (+ 351us) client_negotiation.cc:240] Negotiation successful
0926 03:31:05.839975 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":384,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 03:31:05.843622 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45431
I20260926 03:31:05.844069 629 heartbeater.cc:464] Registering TS with master...
I20260926 03:31:05.844992 629 heartbeater.cc:511] Master 127.0.97.62:45431 requested a full tablet report, sending...
I20260926 03:31:05.847731 440 ts_manager.cc:194] Registered new tserver with Master: ad0e1f2d87a14961a7f3479cc1d4c696 (127.0.97.1:45667)
I20260926 03:31:05.849784 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57925
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:31:05.868042 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45431
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:06.261863 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:06.262229 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:06.262398 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:06.270779 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:06.270910 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:06.271126 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:31:06.281384 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45431
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:06.282974 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:06.284605 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:06.299984 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:06.300005 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:06.299927 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:06.300872 635 server_base.cc:1034] running on GCE node
I20260926 03:31:06.301573 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:06.302644 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:06.303843 635 hybrid_clock.cc:648] HybridClock initialized: now 1790393466303789 us; error 42 us; skew 500 ppm
I20260926 03:31:06.304188 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:06 dist-test-slave-dt7p krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:06.307624 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:31:06.309250 635 webserver.cc:460] Webserver started at http://127.0.97.2:42367/ using document root <none> and password file <none>
I20260926 03:31:06.310002 635 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:06.310102 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:06.310387 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:06.313650 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-1/data/instance:
uuid: "909ac5f086ac4807aa9ce129a4a3aa81"
format_stamp: "Formatted at 2026-09-26 03:31:06 on dist-test-slave-dt7p"
server_key: "4748343077d90101a4231966041d299d"
server_key_iv: "17d38f7e6d643a9772d3914d438c77d1"
server_key_version: "encryptionkey@0"
I20260926 03:31:06.314466 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-1/wal/instance:
uuid: "909ac5f086ac4807aa9ce129a4a3aa81"
format_stamp: "Formatted at 2026-09-26 03:31:06 on dist-test-slave-dt7p"
server_key: "4748343077d90101a4231966041d299d"
server_key_iv: "17d38f7e6d643a9772d3914d438c77d1"
server_key_version: "encryptionkey@0"
I20260926 03:31:06.320245 635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:31:06.324717 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:06.326336 635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:31:06.326478 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-1/wal
uuid: "909ac5f086ac4807aa9ce129a4a3aa81"
format_stamp: "Formatted at 2026-09-26 03:31:06 on dist-test-slave-dt7p"
server_key: "4748343077d90101a4231966041d299d"
server_key_iv: "17d38f7e6d643a9772d3914d438c77d1"
server_key_version: "encryptionkey@0"
I20260926 03:31:06.326671 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:06.351884 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:06.357594 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:06.357892 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:06.359057 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:31:06.360673 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:31:06.360764 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:06.360873 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:31:06.360925 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:06.403872 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40225
I20260926 03:31:06.403908 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40225 every 8 connection(s)
I20260926 03:31:06.405517 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:31:06.407503 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 635
I20260926 03:31:06.407732 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:31:06.408264 388 external_mini_cluster.cc:1428] Setting key 6d621e1a5df32b2b8e09334c2e3703b7
Sep 26 03:31:06 dist-test-slave-dt7p krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 03:31:06.432601 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:06.408703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52969 (local address 127.0.97.62:45431)
0926 03:31:06.409219 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:06.409225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:06.410545 (+ 1320us) server_negotiation.cc:407] Connection header received
0926 03:31:06.412870 (+ 2325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:06.412879 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:06.412988 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:06.413118 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:06.414867 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:06.416314 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:06.420539 (+ 4225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:06.421105 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:06.421230 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:06.426025 (+ 4795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:06.426031 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:06.426034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:06.426072 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:06.428929 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:06.429767 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:06.429773 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:06.429775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:06.429905 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:06.430352 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:06.430363 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:06.430366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:06.430752 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:06.430906 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:06.431794 (+ 888us) server_negotiation.cc:299] Negotiation successful
0926 03:31:06.432096 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":238,"threads_started":1}
I20260926 03:31:06.433856 766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:06.409458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45431 (local address 127.0.97.2:52969)
0926 03:31:06.410352 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:06.410413 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:06.412321 (+ 1908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:06.413389 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:06.413399 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:06.413996 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:06.414685 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:06.414704 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:06.416472 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:06.416479 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:06.420364 (+ 3885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:06.420382 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:06.421348 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:06.421358 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:06.421712 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:06.422601 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:06.422636 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:06.425756 (+ 3120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:06.429071 (+ 3315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:06.429081 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:06.429102 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:06.429631 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:06.430025 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:06.430030 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:06.430040 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:06.430232 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:06.430896 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:06.430904 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:06.431305 (+ 401us) client_negotiation.cc:769] Sending connection context
0926 03:31:06.431722 (+ 417us) client_negotiation.cc:240] Negotiation successful
0926 03:31:06.432375 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":361,"threads_started":1}
I20260926 03:31:06.435544 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45431
I20260926 03:31:06.435959 763 heartbeater.cc:464] Registering TS with master...
I20260926 03:31:06.436941 763 heartbeater.cc:511] Master 127.0.97.62:45431 requested a full tablet report, sending...
I20260926 03:31:06.438710 440 ts_manager.cc:194] Registered new tserver with Master: 909ac5f086ac4807aa9ce129a4a3aa81 (127.0.97.2:40225)
I20260926 03:31:06.439617 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52969
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:31:06.456092 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45431
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:06.850863 769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:06.851207 769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:06.851316 769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 03:31:06.854001 629 heartbeater.cc:503] Master 127.0.97.62:45431 was elected leader, sending a full tablet report...
W20260926 03:31:06.860244 769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:06.860363 769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:06.860496 769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:31:06.871070 769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45431
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:06.872797 769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:06.874439 769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:06.890357 774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:06.890277 777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:06.890379 775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:06.891244 769 server_base.cc:1034] running on GCE node
I20260926 03:31:06.892004 769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:06.893196 769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:06.894456 769 hybrid_clock.cc:648] HybridClock initialized: now 1790393466894370 us; error 75 us; skew 500 ppm
I20260926 03:31:06.894835 769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:06 dist-test-slave-dt7p krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:06.904541 769 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:31:06.906466 769 webserver.cc:460] Webserver started at http://127.0.97.3:41253/ using document root <none> and password file <none>
I20260926 03:31:06.907369 769 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:06.907522 769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:06.907886 769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:06.912427 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-2/data/instance:
uuid: "7a5c3ab004f7461ebc5319721407ca87"
format_stamp: "Formatted at 2026-09-26 03:31:06 on dist-test-slave-dt7p"
server_key: "8d917a2b8a7456e4c651b06a086dc0d6"
server_key_iv: "15871830e2a046f2e494a07ea854ab37"
server_key_version: "encryptionkey@0"
I20260926 03:31:06.913458 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7a5c3ab004f7461ebc5319721407ca87"
format_stamp: "Formatted at 2026-09-26 03:31:06 on dist-test-slave-dt7p"
server_key: "8d917a2b8a7456e4c651b06a086dc0d6"
server_key_iv: "15871830e2a046f2e494a07ea854ab37"
server_key_version: "encryptionkey@0"
I20260926 03:31:06.919894 769 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260926 03:31:06.924104 784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:06.925841 769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260926 03:31:06.926026 769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-2/wal
uuid: "7a5c3ab004f7461ebc5319721407ca87"
format_stamp: "Formatted at 2026-09-26 03:31:06 on dist-test-slave-dt7p"
server_key: "8d917a2b8a7456e4c651b06a086dc0d6"
server_key_iv: "15871830e2a046f2e494a07ea854ab37"
server_key_version: "encryptionkey@0"
I20260926 03:31:06.926249 769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:06.950615 769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:06.956311 769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:06.956702 769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:06.957881 769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:31:06.959442 769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:31:06.959551 769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:06.959702 769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:31:06.959775 769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:07.000777 769 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33489
I20260926 03:31:07.000810 896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33489 every 8 connection(s)
I20260926 03:31:07.002302 769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:31:07.006759 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 769
I20260926 03:31:07.007001 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790393463219689-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:31:07.007602 388 external_mini_cluster.cc:1428] Setting key a7bb5001a05e7cceec7b9a402247eafc
Sep 26 03:31:07 dist-test-slave-dt7p krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:31:07.027351 899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:07.005444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37671 (local address 127.0.97.62:45431)
0926 03:31:07.005965 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:07.005969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:07.007460 (+ 1491us) server_negotiation.cc:407] Connection header received
0926 03:31:07.010637 (+ 3177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:07.010643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:07.010753 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:07.010884 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:07.013014 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:07.013871 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:07.015633 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:07.016076 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:07.016221 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:07.020542 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:07.020549 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:07.020553 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:07.020663 (+ 110us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:07.023705 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:07.024631 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:07.024641 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:07.024644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:07.024782 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:07.025304 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:07.025308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:07.025310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:07.025627 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:07.025796 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:07.026656 (+ 860us) server_negotiation.cc:299] Negotiation successful
0926 03:31:07.026913 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":257,"threads_started":1}
I20260926 03:31:07.028733 900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:07.006201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45431 (local address 127.0.97.3:37671)
0926 03:31:07.007269 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:07.007324 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:07.010315 (+ 2991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:07.011191 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:07.011201 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:07.011872 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:07.012833 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:07.012856 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:07.014041 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:07.014045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:07.015471 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:07.015487 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:07.016323 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:07.016331 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:07.016695 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:07.017514 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:07.017541 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:07.020224 (+ 2683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:07.023891 (+ 3667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:07.023902 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:07.023928 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:07.024461 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:07.024924 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:07.024930 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:07.024934 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:07.025160 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:07.025764 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:07.025783 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:07.026236 (+ 453us) client_negotiation.cc:769] Sending connection context
0926 03:31:07.026553 (+ 317us) client_negotiation.cc:240] Negotiation successful
0926 03:31:07.027248 (+ 695us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":372,"threads_started":1}
I20260926 03:31:07.030427 897 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45431
I20260926 03:31:07.030851 897 heartbeater.cc:464] Registering TS with master...
I20260926 03:31:07.031965 897 heartbeater.cc:511] Master 127.0.97.62:45431 requested a full tablet report, sending...
I20260926 03:31:07.033717 440 ts_manager.cc:194] Registered new tserver with Master: 7a5c3ab004f7461ebc5319721407ca87 (127.0.97.3:33489)
I20260926 03:31:07.034610 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37671
I20260926 03:31:07.037806 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 03:31:07 dist-test-slave-dt7p krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393467, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 03:31:07 dist-test-slave-dt7p krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393467, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:31:07.092545 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:07.073988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60712 (local address 127.0.97.1:45667)
0926 03:31:07.074887 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:07.074900 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:07.074942 (+ 42us) server_negotiation.cc:407] Connection header received
0926 03:31:07.075053 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:07.075065 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:07.075437 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:07.075658 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:07.077198 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:07.078922 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:07.080291 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:07.081351 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:07.081483 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:07.086398 (+ 4915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:07.086409 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:07.086429 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:07.086464 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:07.089801 (+ 3337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:07.090521 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:07.090531 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:07.090534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:07.090682 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:07.091036 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:07.091040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:07.091042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:07.091492 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:07.091744 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:07.091965 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 03:31:07.092226 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":478,"threads_started":1}
W20260926 03:31:07.094182 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60712
Sep 26 03:31:07 dist-test-slave-dt7p krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 03:31:07 dist-test-slave-dt7p krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:31:07.142622 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:07.127841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60728 (local address 127.0.97.1:45667)
0926 03:31:07.128118 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:07.128126 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:07.128317 (+ 191us) server_negotiation.cc:407] Connection header received
0926 03:31:07.128491 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:07.128496 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:07.128643 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:07.128740 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:07.130160 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:07.131081 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:07.132540 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:07.133026 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:07.133127 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:07.137159 (+ 4032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:07.137164 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:07.137168 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:07.137211 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:07.140382 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:07.141011 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:07.141015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:07.141017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:07.141146 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:07.141466 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:07.141469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:07.141471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:07.141698 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:07.141878 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:07.142108 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 03:31:07.142284 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260926 03:31:07.144315 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 501
I20260926 03:31:07.157051 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:07.145361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60742 (local address 127.0.97.1:45667)
0926 03:31:07.145525 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:07.145529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:07.145953 (+ 424us) server_negotiation.cc:407] Connection header received
0926 03:31:07.146075 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:07.146079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:07.146133 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:07.146210 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:07.147383 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:07.148118 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:07.149481 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:07.149989 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:07.150081 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:07.152204 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:07.152209 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:07.152212 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:07.152241 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:07.154816 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:07.155408 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:07.155412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:07.155414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:07.155535 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:07.155825 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:07.155828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:07.155830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:07.156058 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:07.156237 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:07.156455 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 03:31:07.156665 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 03:31:07.158044 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:31:07.248468 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 501
I20260926 03:31:07.266595 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 635
Sep 26 03:31:07 dist-test-slave-dt7p krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:31:07.283782 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:07.267671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34692 (local address 127.0.97.2:40225)
0926 03:31:07.268352 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:07.268361 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:07.268399 (+ 38us) server_negotiation.cc:407] Connection header received
0926 03:31:07.268475 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:07.268483 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:07.268863 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:07.269025 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:07.270331 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:07.271499 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:07.272903 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:07.273532 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:07.273634 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:07.277974 (+ 4340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:07.277986 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:07.278005 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:07.278050 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:07.281087 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:07.281701 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:07.281709 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:07.281712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:07.281836 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:07.282206 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:07.282209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:07.282211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:07.282579 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:07.282819 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:07.283105 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 03:31:07.283328 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":373,"threads_started":1}
I20260926 03:31:07.285588 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:31:07.374624 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 635
I20260926 03:31:07.400069 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 769
Sep 26 03:31:07 dist-test-slave-dt7p krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:31:07.418603 914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:07.401407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57842 (local address 127.0.97.3:33489)
0926 03:31:07.402168 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:07.402177 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:07.402213 (+ 36us) server_negotiation.cc:407] Connection header received
0926 03:31:07.402299 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:07.402308 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:07.402613 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:07.402773 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:07.404217 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:07.405525 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:07.406998 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:07.407658 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:07.407773 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:07.412756 (+ 4983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:07.412768 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:07.412796 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:07.412836 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:07.415687 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:07.416341 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:07.416354 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:07.416362 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:07.416486 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:07.416920 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:07.416924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:07.416926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:07.417346 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:07.417618 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:07.417901 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 03:31:07.418133 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":398,"threads_started":1}
I20260926 03:31:07.420389 892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:31:07.508781 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 769
I20260926 03:31:07.526350 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 408
I20260926 03:31:07.528005 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:31:07.644701 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 408
2026-09-26T03:31:07Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4382 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:31:07 dist-test-slave-dt7p krb5kdc[919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:31:07 dist-test-slave-dt7p krb5kdc[919](info): set up 2 sockets
Sep 26 03:31:07 dist-test-slave-dt7p krb5kdc[919](info): commencing operation
krb5kdc: starting...
W20260926 03:31:08.732854 388 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.002s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:31:08 dist-test-slave-dt7p krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:31:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:31:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:31:08.932675 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46823
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46823
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:09.313488 932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:09.313877 932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:09.314042 932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:09.322244 932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:31:09.322389 932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:09.322441 932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:09.322559 932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:31:09.322629 932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:31:09.333954 932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:09.335793 932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:09.337604 932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:09.351760 940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:09.351833 938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:09.352006 937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:09.352643 932 server_base.cc:1034] running on GCE node
I20260926 03:31:09.353796 932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:09.355597 932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:09.356846 932 hybrid_clock.cc:648] HybridClock initialized: now 1790393469356778 us; error 59 us; skew 500 ppm
I20260926 03:31:09.357281 932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:09 dist-test-slave-dt7p krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393469, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:09.361071 932 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:31:09.362989 932 webserver.cc:460] Webserver started at http://127.0.97.62:34895/ using document root <none> and password file <none>
I20260926 03:31:09.363893 932 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:09.364048 932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:09.364429 932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:09.368147 932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "8783d2300e48448eb64666401c2f3e53"
format_stamp: "Formatted at 2026-09-26 03:31:09 on dist-test-slave-dt7p"
server_key: "12f9e4f95a1305cb66df88777e7f77d0"
server_key_iv: "b065ca327554d62e6b71be6f4220126a"
server_key_version: "encryptionkey@0"
I20260926 03:31:09.369109 932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "8783d2300e48448eb64666401c2f3e53"
format_stamp: "Formatted at 2026-09-26 03:31:09 on dist-test-slave-dt7p"
server_key: "12f9e4f95a1305cb66df88777e7f77d0"
server_key_iv: "b065ca327554d62e6b71be6f4220126a"
server_key_version: "encryptionkey@0"
I20260926 03:31:09.375653 932 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260926 03:31:09.380116 947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:09.382120 932 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 03:31:09.382377 932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "8783d2300e48448eb64666401c2f3e53"
format_stamp: "Formatted at 2026-09-26 03:31:09 on dist-test-slave-dt7p"
server_key: "12f9e4f95a1305cb66df88777e7f77d0"
server_key_iv: "b065ca327554d62e6b71be6f4220126a"
server_key_version: "encryptionkey@0"
I20260926 03:31:09.382650 932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:09.404848 932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:09.410611 932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:09.410970 932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:09.432819 932 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46823
I20260926 03:31:09.432852 998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46823 every 8 connection(s)
I20260926 03:31:09.434599 932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:31:09.439507 999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:09.444394 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 932
I20260926 03:31:09.444717 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:31:09.445664 388 external_mini_cluster.cc:1428] Setting key 38d3ced370392fe14cf5a25d54555dfa
I20260926 03:31:09.454121 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8783d2300e48448eb64666401c2f3e53: Bootstrap starting.
I20260926 03:31:09.458950 999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8783d2300e48448eb64666401c2f3e53: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:09.460551 999 log.cc:824] T 00000000000000000000000000000000 P 8783d2300e48448eb64666401c2f3e53: Log is configured to *not* fsync() on all Append() calls
Sep 26 03:31:09 dist-test-slave-dt7p krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:31:09.464882 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8783d2300e48448eb64666401c2f3e53: No bootstrap required, opened a new log
I20260926 03:31:09.476497 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:09.448418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51056 (local address 127.0.97.62:46823)
0926 03:31:09.449470 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:09.449486 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:09.449539 (+ 53us) server_negotiation.cc:407] Connection header received
0926 03:31:09.452188 (+ 2649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:09.452221 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:09.452846 (+ 625us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:09.453270 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:09.455443 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:09.457062 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:09.459024 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:09.459904 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:09.460056 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:09.465765 (+ 5709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:09.465780 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:09.465807 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:09.465874 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:09.470635 (+ 4761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:09.471477 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:09.471491 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:09.471503 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:09.471739 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:09.472131 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:09.472136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:09.472140 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:09.473082 (+ 942us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:09.473421 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:09.473989 (+ 568us) server_negotiation.cc:299] Negotiation successful
0926 03:31:09.474578 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":798,"thread_start_us":466,"threads_started":1}
I20260926 03:31:09.477411 999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8783d2300e48448eb64666401c2f3e53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8783d2300e48448eb64666401c2f3e53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46823 } }
I20260926 03:31:09.477878 999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8783d2300e48448eb64666401c2f3e53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:09.478011 999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8783d2300e48448eb64666401c2f3e53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8783d2300e48448eb64666401c2f3e53, State: Initialized, Role: FOLLOWER
I20260926 03:31:09.478817 999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8783d2300e48448eb64666401c2f3e53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8783d2300e48448eb64666401c2f3e53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46823 } }
I20260926 03:31:09.479131 999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8783d2300e48448eb64666401c2f3e53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:31:09.479293 999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8783d2300e48448eb64666401c2f3e53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:31:09.479498 999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8783d2300e48448eb64666401c2f3e53 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:09.482249 999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8783d2300e48448eb64666401c2f3e53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8783d2300e48448eb64666401c2f3e53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46823 } }
I20260926 03:31:09.482911 999 leader_election.cc:302] T 00000000000000000000000000000000 P 8783d2300e48448eb64666401c2f3e53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8783d2300e48448eb64666401c2f3e53; no voters:
I20260926 03:31:09.483642 999 leader_election.cc:288] T 00000000000000000000000000000000 P 8783d2300e48448eb64666401c2f3e53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:31:09.483829 1004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8783d2300e48448eb64666401c2f3e53 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:31:09.484679 1004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8783d2300e48448eb64666401c2f3e53 [term 1 LEADER]: Becoming Leader. State: Replica: 8783d2300e48448eb64666401c2f3e53, State: Running, Role: LEADER
I20260926 03:31:09.485292 1004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8783d2300e48448eb64666401c2f3e53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8783d2300e48448eb64666401c2f3e53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46823 } }
I20260926 03:31:09.486402 999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8783d2300e48448eb64666401c2f3e53 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:31:09.488798 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8783d2300e48448eb64666401c2f3e53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8783d2300e48448eb64666401c2f3e53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8783d2300e48448eb64666401c2f3e53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46823 } } }
I20260926 03:31:09.489235 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8783d2300e48448eb64666401c2f3e53 [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:09.489989 1006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8783d2300e48448eb64666401c2f3e53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8783d2300e48448eb64666401c2f3e53. Latest consensus state: current_term: 1 leader_uuid: "8783d2300e48448eb64666401c2f3e53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8783d2300e48448eb64666401c2f3e53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46823 } } }
I20260926 03:31:09.490269 1006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8783d2300e48448eb64666401c2f3e53 [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:09.490386 1011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:31:09.499564 1011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:31:09.511233 1011 catalog_manager.cc:1347] Generated new cluster ID: aa2fd670b9ba4bfd8bdb4bcb46a73280
I20260926 03:31:09.511449 1011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:31:09.545717 1011 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:31:09.547559 1011 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:31:09.555864 1011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8783d2300e48448eb64666401c2f3e53: Generated new TSK 0
I20260926 03:31:09.557085 1011 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:31:09.621405 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46823
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:10.009387 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:10.009793 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:10.009955 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:10.017673 1025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:10.017817 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:10.017947 1025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:31:10.028815 1025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46823
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:10.030678 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:10.032351 1025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:10.047890 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:10.047919 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:10.047995 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:10.054382 1025 server_base.cc:1034] running on GCE node
I20260926 03:31:10.055200 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:10.056645 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:10.057904 1025 hybrid_clock.cc:648] HybridClock initialized: now 1790393470057832 us; error 51 us; skew 500 ppm
I20260926 03:31:10.058451 1025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:10 dist-test-slave-dt7p krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:10.062759 1025 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:31:10.064481 1025 webserver.cc:460] Webserver started at http://127.0.97.1:43659/ using document root <none> and password file <none>
I20260926 03:31:10.065429 1025 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:10.065541 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:10.065845 1025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:10.069445 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-0/data/instance:
uuid: "e8f3740271394ed181a72f1c7a0dee5a"
format_stamp: "Formatted at 2026-09-26 03:31:10 on dist-test-slave-dt7p"
server_key: "08cba62f6d4a80c2694d93ff48e2e327"
server_key_iv: "7b108a00247edb0b8faf34564f3856b8"
server_key_version: "encryptionkey@0"
I20260926 03:31:10.070338 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e8f3740271394ed181a72f1c7a0dee5a"
format_stamp: "Formatted at 2026-09-26 03:31:10 on dist-test-slave-dt7p"
server_key: "08cba62f6d4a80c2694d93ff48e2e327"
server_key_iv: "7b108a00247edb0b8faf34564f3856b8"
server_key_version: "encryptionkey@0"
I20260926 03:31:10.076462 1025 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 03:31:10.080716 1040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:10.082834 1025 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 03:31:10.083040 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-0/wal
uuid: "e8f3740271394ed181a72f1c7a0dee5a"
format_stamp: "Formatted at 2026-09-26 03:31:10 on dist-test-slave-dt7p"
server_key: "08cba62f6d4a80c2694d93ff48e2e327"
server_key_iv: "7b108a00247edb0b8faf34564f3856b8"
server_key_version: "encryptionkey@0"
I20260926 03:31:10.083273 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:10.098033 1025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:10.102937 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:10.103245 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:10.104339 1025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:31:10.105975 1025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:31:10.106053 1025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:10.106189 1025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:31:10.106257 1025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:10.142832 1025 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42983
I20260926 03:31:10.142834 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42983 every 8 connection(s)
I20260926 03:31:10.144968 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:31:10.153227 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 1025
I20260926 03:31:10.153486 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:31:10.154184 388 external_mini_cluster.cc:1428] Setting key 22e18c054760aae84367b9d562c8c90d
Sep 26 03:31:10 dist-test-slave-dt7p krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:31:10.168735 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:10.148106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59969 (local address 127.0.97.62:46823)
0926 03:31:10.148695 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:10.148702 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:10.149944 (+ 1242us) server_negotiation.cc:407] Connection header received
0926 03:31:10.151954 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:10.151959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:10.152051 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:10.152163 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:10.154173 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:10.155313 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:10.157493 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:10.158303 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:10.158473 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:10.162523 (+ 4050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:10.162528 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:10.162532 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:10.162566 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:10.165433 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:10.166239 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:10.166243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:10.166246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:10.166367 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:10.166829 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:10.166841 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:10.166843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:10.167116 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:10.167283 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:10.168054 (+ 771us) server_negotiation.cc:299] Negotiation successful
0926 03:31:10.168302 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":277,"threads_started":1}
I20260926 03:31:10.169977 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:10.148919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46823 (local address 127.0.97.1:59969)
0926 03:31:10.149760 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:10.149807 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:10.151634 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:10.152432 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:10.152442 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:10.153170 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:10.153979 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:10.154001 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:10.155517 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:10.155522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:10.157340 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:10.157352 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:10.158536 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:10.158540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:10.158765 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:10.159573 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:10.159599 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:10.162262 (+ 2663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:10.165597 (+ 3335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:10.165607 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:10.165636 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:10.166093 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:10.166479 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:10.166486 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:10.166489 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:10.166702 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:10.167246 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:10.167260 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:10.167655 (+ 395us) client_negotiation.cc:769] Sending connection context
0926 03:31:10.167956 (+ 301us) client_negotiation.cc:240] Negotiation successful
0926 03:31:10.168570 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":327,"threads_started":1}
I20260926 03:31:10.172101 1153 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46823
I20260926 03:31:10.172500 1153 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 03:31:10.173480 1153 heartbeater.cc:511] Master 127.0.97.62:46823 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 03:31:10.176358 964 ts_manager.cc:194] Registered new tserver with Master: e8f3740271394ed181a72f1c7a0dee5a (127.0.97.1:42983)
I20260926 03:31:10.178115 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59969
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:31:10.200659 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46823
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:10.599593 1159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:10.599956 1159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:10.600116 1159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:10.607985 1159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:10.608121 1159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:10.608258 1159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:31:10.618980 1159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46823
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:10.620697 1159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:10.622359 1159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:10.637894 1164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:10.637825 1165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:10.637866 1167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:10.639134 1159 server_base.cc:1034] running on GCE node
I20260926 03:31:10.639999 1159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:10.641305 1159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:10.642566 1159 hybrid_clock.cc:648] HybridClock initialized: now 1790393470642524 us; error 40 us; skew 500 ppm
I20260926 03:31:10.643003 1159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:10 dist-test-slave-dt7p krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:10.646994 1159 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:31:10.648859 1159 webserver.cc:460] Webserver started at http://127.0.97.2:39995/ using document root <none> and password file <none>
I20260926 03:31:10.649717 1159 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:10.649828 1159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:10.650135 1159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:10.653712 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-1/data/instance:
uuid: "a5c133bdea074d15a00d030f58842177"
format_stamp: "Formatted at 2026-09-26 03:31:10 on dist-test-slave-dt7p"
server_key: "c0af11d6acfdd7038d6e2a3973819249"
server_key_iv: "f80ecd587013ccdf7a6962e8e8f56268"
server_key_version: "encryptionkey@0"
I20260926 03:31:10.654560 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a5c133bdea074d15a00d030f58842177"
format_stamp: "Formatted at 2026-09-26 03:31:10 on dist-test-slave-dt7p"
server_key: "c0af11d6acfdd7038d6e2a3973819249"
server_key_iv: "f80ecd587013ccdf7a6962e8e8f56268"
server_key_version: "encryptionkey@0"
I20260926 03:31:10.660739 1159 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260926 03:31:10.665506 1174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:10.667514 1159 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 03:31:10.667701 1159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-1/wal
uuid: "a5c133bdea074d15a00d030f58842177"
format_stamp: "Formatted at 2026-09-26 03:31:10 on dist-test-slave-dt7p"
server_key: "c0af11d6acfdd7038d6e2a3973819249"
server_key_iv: "f80ecd587013ccdf7a6962e8e8f56268"
server_key_version: "encryptionkey@0"
I20260926 03:31:10.667917 1159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:10.706334 1159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:10.711476 1159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:10.711740 1159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:10.712883 1159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:31:10.714365 1159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:31:10.714440 1159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:10.714536 1159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:31:10.714582 1159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:10.751639 1159 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38023
I20260926 03:31:10.751688 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38023 every 8 connection(s)
I20260926 03:31:10.753419 1159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:31:10.763634 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 1159
I20260926 03:31:10.763880 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:31:10.764389 388 external_mini_cluster.cc:1428] Setting key ea853bfc86d7fd29a744001359abb863
Sep 26 03:31:10 dist-test-slave-dt7p krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:31:10.777501 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:10.756809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53331 (local address 127.0.97.62:46823)
0926 03:31:10.757322 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:10.757327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:10.758564 (+ 1237us) server_negotiation.cc:407] Connection header received
0926 03:31:10.760654 (+ 2090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:10.760660 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:10.760769 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:10.760912 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:10.762816 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:10.763919 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:10.766023 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:10.766674 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:10.766817 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:10.771094 (+ 4277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:10.771099 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:10.771103 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:10.771140 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:10.774084 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:10.774899 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:10.774905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:10.774907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:10.775036 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:10.775490 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:10.775500 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:10.775502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:10.775805 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:10.775988 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:10.776844 (+ 856us) server_negotiation.cc:299] Negotiation successful
0926 03:31:10.777096 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":274,"threads_started":1}
I20260926 03:31:10.778894 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:10.757540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46823 (local address 127.0.97.2:53331)
0926 03:31:10.758350 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:10.758421 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:10.760300 (+ 1879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:10.761205 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:10.761216 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:10.761867 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:10.762614 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:10.762636 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:10.764086 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:10.764090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:10.765883 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:10.765896 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:10.766929 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:10.766935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:10.767202 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:10.768054 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:10.768085 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:10.770845 (+ 2760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:10.774230 (+ 3385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:10.774241 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:10.774263 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:10.774751 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:10.775146 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:10.775152 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:10.775156 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:10.775360 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:10.775946 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:10.775956 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:10.776400 (+ 444us) client_negotiation.cc:769] Sending connection context
0926 03:31:10.776752 (+ 352us) client_negotiation.cc:240] Negotiation successful
0926 03:31:10.777415 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":327,"threads_started":1}
I20260926 03:31:10.780807 1287 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46823
I20260926 03:31:10.781167 1287 heartbeater.cc:464] Registering TS with master...
I20260926 03:31:10.782027 1287 heartbeater.cc:511] Master 127.0.97.62:46823 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 03:31:10.783825 964 ts_manager.cc:194] Registered new tserver with Master: a5c133bdea074d15a00d030f58842177 (127.0.97.2:38023)
I20260926 03:31:10.784862 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53331
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:31:10.812711 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46823
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:31:11.181862 1153 heartbeater.cc:503] Master 127.0.97.62:46823 was elected leader, sending a full tablet report...
W20260926 03:31:11.186749 1293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:11.187177 1293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:11.187394 1293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:11.195466 1293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:11.195596 1293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:11.195797 1293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:31:11.206658 1293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46823
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:11.208386 1293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:11.210103 1293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:11.226361 1299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:11.226414 1301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:11.226437 1298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:11.230312 1293 server_base.cc:1034] running on GCE node
I20260926 03:31:11.231170 1293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:11.232582 1293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:11.233847 1293 hybrid_clock.cc:648] HybridClock initialized: now 1790393471233780 us; error 62 us; skew 500 ppm
I20260926 03:31:11.234241 1293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:11 dist-test-slave-dt7p krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:11.238410 1293 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:31:11.240799 1293 webserver.cc:460] Webserver started at http://127.0.97.3:36285/ using document root <none> and password file <none>
I20260926 03:31:11.241781 1293 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:11.241950 1293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:11.242342 1293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:11.246023 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-2/data/instance:
uuid: "a0d2886d3bb34a179c070a4986291e68"
format_stamp: "Formatted at 2026-09-26 03:31:11 on dist-test-slave-dt7p"
server_key: "d3ae5b24b31d0e6e406567f6ab22def7"
server_key_iv: "d2fd50e7a197fa486ea387f510e8f32c"
server_key_version: "encryptionkey@0"
I20260926 03:31:11.246896 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a0d2886d3bb34a179c070a4986291e68"
format_stamp: "Formatted at 2026-09-26 03:31:11 on dist-test-slave-dt7p"
server_key: "d3ae5b24b31d0e6e406567f6ab22def7"
server_key_iv: "d2fd50e7a197fa486ea387f510e8f32c"
server_key_version: "encryptionkey@0"
I20260926 03:31:11.253111 1293 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 03:31:11.257104 1308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:11.258790 1293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:31:11.258992 1293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-2/wal
uuid: "a0d2886d3bb34a179c070a4986291e68"
format_stamp: "Formatted at 2026-09-26 03:31:11 on dist-test-slave-dt7p"
server_key: "d3ae5b24b31d0e6e406567f6ab22def7"
server_key_iv: "d2fd50e7a197fa486ea387f510e8f32c"
server_key_version: "encryptionkey@0"
I20260926 03:31:11.259285 1293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:11.273969 1293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:11.278985 1293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:11.279315 1293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:11.280474 1293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:31:11.282024 1293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:31:11.282099 1293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:11.282191 1293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:31:11.282235 1293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:11.316726 1293 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33583
I20260926 03:31:11.316768 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33583 every 8 connection(s)
I20260926 03:31:11.318305 1293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:31:11.323073 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 1293
I20260926 03:31:11.323305 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790393463219689-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:31:11.323915 388 external_mini_cluster.cc:1428] Setting key f984710e993724446a4f4ddc8108f4dd
Sep 26 03:31:11 dist-test-slave-dt7p krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:31:11.341048 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:11.321724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60363 (local address 127.0.97.62:46823)
0926 03:31:11.322200 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:11.322205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:11.323698 (+ 1493us) server_negotiation.cc:407] Connection header received
0926 03:31:11.325914 (+ 2216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:11.325920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:11.326019 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:11.326158 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:11.328071 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:11.328977 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:11.330661 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:11.331100 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:11.331240 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:11.335034 (+ 3794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:11.335039 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:11.335043 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:11.335079 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:11.337871 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:11.338606 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:11.338614 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:11.338617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:11.338743 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:11.339164 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:11.339172 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:11.339176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:11.339469 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:11.339630 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:11.340380 (+ 750us) server_negotiation.cc:299] Negotiation successful
0926 03:31:11.340631 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":219,"threads_started":1}
I20260926 03:31:11.342276 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:11.322492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46823 (local address 127.0.97.3:60363)
0926 03:31:11.323507 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:11.323562 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:11.325596 (+ 2034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:11.326445 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:11.326456 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:11.327174 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:11.327897 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:11.327916 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:11.329130 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:11.329137 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:11.330520 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:11.330531 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:11.331332 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:11.331340 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:11.331594 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:11.332367 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:11.332391 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:11.334812 (+ 2421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:11.338018 (+ 3206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:11.338028 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:11.338049 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:11.338484 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:11.338849 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:11.338855 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:11.338859 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:11.339053 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:11.339597 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:11.339607 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:11.339985 (+ 378us) client_negotiation.cc:769] Sending connection context
0926 03:31:11.340280 (+ 295us) client_negotiation.cc:240] Negotiation successful
0926 03:31:11.340914 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":383,"threads_started":1}
I20260926 03:31:11.343803 1421 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46823
I20260926 03:31:11.344174 1421 heartbeater.cc:464] Registering TS with master...
I20260926 03:31:11.345156 1421 heartbeater.cc:511] Master 127.0.97.62:46823 requested a full tablet report, sending...
I20260926 03:31:11.346885 964 ts_manager.cc:194] Registered new tserver with Master: a0d2886d3bb34a179c070a4986291e68 (127.0.97.3:33583)
I20260926 03:31:11.347842 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60363
I20260926 03:31:11.352283 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:31:11.375643 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:11.360012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51060 (local address 127.0.97.62:46823)
0926 03:31:11.360252 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:11.360263 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:11.360907 (+ 644us) server_negotiation.cc:407] Connection header received
0926 03:31:11.361079 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:11.361083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:11.361160 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:11.361280 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:11.362973 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:11.363889 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:11.365430 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:11.365879 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:11.365978 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:11.369399 (+ 3421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:11.369404 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:11.369409 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:11.369447 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:11.372586 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:11.373383 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:11.373388 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:11.373391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:11.373524 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:11.373982 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:11.373987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:11.373991 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:11.374390 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:11.374662 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:11.375038 (+ 376us) server_negotiation.cc:299] Negotiation successful
0926 03:31:11.375278 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:31:11.383560 964 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:31:11.391651 964 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:31:11.433463 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:11.408420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56520 (local address 127.0.97.3:33583)
0926 03:31:11.420053 (+ 11633us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:11.420062 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:11.420104 (+ 42us) server_negotiation.cc:407] Connection header received
0926 03:31:11.420231 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:11.420243 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:11.420630 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:11.421494 (+ 864us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:31:11.421968 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:11.425571 (+ 3603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:11.430265 (+ 4694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:11.431553 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:11.431716 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:11.432021 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:11.432843 (+ 822us) server_negotiation.cc:299] Negotiation successful
0926 03:31:11.433081 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11430,"thread_start_us":11222,"threads_started":1}
I20260926 03:31:11.434159 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:11.407298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33583 (local address 127.0.0.1:56520)
0926 03:31:11.419092 (+ 11794us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:11.419168 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:11.419404 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:11.420939 (+ 1535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:11.420954 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:11.421027 (+ 73us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:31:11.421707 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:11.421726 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:11.425788 (+ 4062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:11.425796 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:11.430078 (+ 4282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:11.430092 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:11.431755 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:11.431761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:11.431954 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:11.432013 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 03:31:11.432153 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 03:31:11.432397 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11387,"mutex_wait_us":419,"spinlock_wait_cycles":1920,"thread_start_us":11206,"threads_started":1}
I20260926 03:31:11.436502 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:11.407806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42983 (local address 127.0.0.1:44272)
0926 03:31:11.419992 (+ 12186us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:11.420010 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:11.420140 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:11.420963 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:11.420969 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:11.421043 (+ 74us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:31:11.422247 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:11.422273 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:11.425607 (+ 3334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:11.425615 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:11.430701 (+ 5086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:11.430716 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:11.434814 (+ 4098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:11.434823 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:11.435003 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:11.435037 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 03:31:11.435131 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 03:31:11.436035 (+ 904us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12068,"mutex_wait_us":61,"spinlock_wait_cycles":21504,"thread_start_us":12136,"threads_started":1}
I20260926 03:31:11.437485 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:11.407917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44272 (local address 127.0.97.1:42983)
0926 03:31:11.418491 (+ 10574us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:11.418505 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:11.420286 (+ 1781us) server_negotiation.cc:407] Connection header received
0926 03:31:11.420444 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:11.420455 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:11.420732 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:11.422482 (+ 1750us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:31:11.422794 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:11.425415 (+ 2621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:11.430866 (+ 5451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:11.432961 (+ 2095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:11.433109 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:11.434654 (+ 1545us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:11.436890 (+ 2236us) server_negotiation.cc:299] Negotiation successful
0926 03:31:11.437101 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10337,"thread_start_us":10152,"threads_started":1}
I20260926 03:31:11.438550 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:11.420610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38023 (local address 127.0.0.1:49664)
0926 03:31:11.428920 (+ 8310us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:11.428939 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:11.429076 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:11.429917 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:11.429923 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:11.429955 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:31:11.430424 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:11.430433 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:11.432342 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:11.432348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:11.434532 (+ 2184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:11.434546 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:11.437676 (+ 3130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:11.437682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:11.437873 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:11.437906 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 03:31:11.437999 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 03:31:11.438155 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8180,"mutex_wait_us":3,"thread_start_us":8018,"threads_started":1}
I20260926 03:31:11.440323 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:11.420735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49664 (local address 127.0.97.2:38023)
0926 03:31:11.428594 (+ 7859us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:11.428610 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:11.429053 (+ 443us) server_negotiation.cc:407] Connection header received
0926 03:31:11.429280 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:11.429290 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:11.429654 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:11.429915 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:31:11.430595 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:11.432152 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:11.434732 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:11.437172 (+ 2440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:11.437308 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:11.437522 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:11.438869 (+ 1347us) server_negotiation.cc:299] Negotiation successful
0926 03:31:11.439762 (+ 893us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7580,"thread_start_us":7368,"threads_started":1}
I20260926 03:31:11.440060 1356 tablet_service.cc:1467] Processing CreateTablet for tablet d1464bf2ac73469e9a9b18135bf4a311 (DEFAULT_TABLE table=test-table [id=8421c7f5e9b74963af2826916e10b78a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:31:11.442165 1356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1464bf2ac73469e9a9b18135bf4a311. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:11.442353 1088 tablet_service.cc:1467] Processing CreateTablet for tablet d1464bf2ac73469e9a9b18135bf4a311 (DEFAULT_TABLE table=test-table [id=8421c7f5e9b74963af2826916e10b78a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:31:11.444001 1088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1464bf2ac73469e9a9b18135bf4a311. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:11.444169 1222 tablet_service.cc:1467] Processing CreateTablet for tablet d1464bf2ac73469e9a9b18135bf4a311 (DEFAULT_TABLE table=test-table [id=8421c7f5e9b74963af2826916e10b78a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:31:11.445475 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1464bf2ac73469e9a9b18135bf4a311. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:11.466502 1440 tablet_bootstrap.cc:492] T d1464bf2ac73469e9a9b18135bf4a311 P a0d2886d3bb34a179c070a4986291e68: Bootstrap starting.
I20260926 03:31:11.470893 1441 tablet_bootstrap.cc:492] T d1464bf2ac73469e9a9b18135bf4a311 P a5c133bdea074d15a00d030f58842177: Bootstrap starting.
I20260926 03:31:11.472388 1442 tablet_bootstrap.cc:492] T d1464bf2ac73469e9a9b18135bf4a311 P e8f3740271394ed181a72f1c7a0dee5a: Bootstrap starting.
I20260926 03:31:11.475312 1441 tablet_bootstrap.cc:654] T d1464bf2ac73469e9a9b18135bf4a311 P a5c133bdea074d15a00d030f58842177: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:11.476235 1442 tablet_bootstrap.cc:654] T d1464bf2ac73469e9a9b18135bf4a311 P e8f3740271394ed181a72f1c7a0dee5a: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:11.476226 1440 tablet_bootstrap.cc:654] T d1464bf2ac73469e9a9b18135bf4a311 P a0d2886d3bb34a179c070a4986291e68: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:11.477772 1441 log.cc:824] T d1464bf2ac73469e9a9b18135bf4a311 P a5c133bdea074d15a00d030f58842177: Log is configured to *not* fsync() on all Append() calls
I20260926 03:31:11.477775 1442 log.cc:824] T d1464bf2ac73469e9a9b18135bf4a311 P e8f3740271394ed181a72f1c7a0dee5a: Log is configured to *not* fsync() on all Append() calls
I20260926 03:31:11.478039 1440 log.cc:824] T d1464bf2ac73469e9a9b18135bf4a311 P a0d2886d3bb34a179c070a4986291e68: Log is configured to *not* fsync() on all Append() calls
I20260926 03:31:11.481366 1441 tablet_bootstrap.cc:492] T d1464bf2ac73469e9a9b18135bf4a311 P a5c133bdea074d15a00d030f58842177: No bootstrap required, opened a new log
I20260926 03:31:11.481419 1442 tablet_bootstrap.cc:492] T d1464bf2ac73469e9a9b18135bf4a311 P e8f3740271394ed181a72f1c7a0dee5a: No bootstrap required, opened a new log
I20260926 03:31:11.481417 1440 tablet_bootstrap.cc:492] T d1464bf2ac73469e9a9b18135bf4a311 P a0d2886d3bb34a179c070a4986291e68: No bootstrap required, opened a new log
I20260926 03:31:11.481683 1441 ts_tablet_manager.cc:1397] T d1464bf2ac73469e9a9b18135bf4a311 P a5c133bdea074d15a00d030f58842177: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20260926 03:31:11.481695 1442 ts_tablet_manager.cc:1397] T d1464bf2ac73469e9a9b18135bf4a311 P e8f3740271394ed181a72f1c7a0dee5a: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260926 03:31:11.481695 1440 ts_tablet_manager.cc:1397] T d1464bf2ac73469e9a9b18135bf4a311 P a0d2886d3bb34a179c070a4986291e68: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.006s
I20260926 03:31:11.490192 1441 raft_consensus.cc:348] T d1464bf2ac73469e9a9b18135bf4a311 P a5c133bdea074d15a00d030f58842177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d2886d3bb34a179c070a4986291e68" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33583 } } peers { permanent_uuid: "e8f3740271394ed181a72f1c7a0dee5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42983 } } peers { permanent_uuid: "a5c133bdea074d15a00d030f58842177" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38023 } }
I20260926 03:31:11.490682 1441 raft_consensus.cc:374] T d1464bf2ac73469e9a9b18135bf4a311 P a5c133bdea074d15a00d030f58842177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:11.490808 1441 raft_consensus.cc:729] T d1464bf2ac73469e9a9b18135bf4a311 P a5c133bdea074d15a00d030f58842177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5c133bdea074d15a00d030f58842177, State: Initialized, Role: FOLLOWER
I20260926 03:31:11.492265 1441 consensus_queue.cc:260] T d1464bf2ac73469e9a9b18135bf4a311 P a5c133bdea074d15a00d030f58842177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d2886d3bb34a179c070a4986291e68" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33583 } } peers { permanent_uuid: "e8f3740271394ed181a72f1c7a0dee5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42983 } } peers { permanent_uuid: "a5c133bdea074d15a00d030f58842177" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38023 } }
I20260926 03:31:11.494508 1440 raft_consensus.cc:348] T d1464bf2ac73469e9a9b18135bf4a311 P a0d2886d3bb34a179c070a4986291e68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d2886d3bb34a179c070a4986291e68" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33583 } } peers { permanent_uuid: "e8f3740271394ed181a72f1c7a0dee5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42983 } } peers { permanent_uuid: "a5c133bdea074d15a00d030f58842177" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38023 } }
I20260926 03:31:11.494509 1442 raft_consensus.cc:348] T d1464bf2ac73469e9a9b18135bf4a311 P e8f3740271394ed181a72f1c7a0dee5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d2886d3bb34a179c070a4986291e68" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33583 } } peers { permanent_uuid: "e8f3740271394ed181a72f1c7a0dee5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42983 } } peers { permanent_uuid: "a5c133bdea074d15a00d030f58842177" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38023 } }
I20260926 03:31:11.495039 1442 raft_consensus.cc:374] T d1464bf2ac73469e9a9b18135bf4a311 P e8f3740271394ed181a72f1c7a0dee5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:11.495060 1440 raft_consensus.cc:374] T d1464bf2ac73469e9a9b18135bf4a311 P a0d2886d3bb34a179c070a4986291e68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:11.495162 1440 raft_consensus.cc:729] T d1464bf2ac73469e9a9b18135bf4a311 P a0d2886d3bb34a179c070a4986291e68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d2886d3bb34a179c070a4986291e68, State: Initialized, Role: FOLLOWER
I20260926 03:31:11.495222 1442 raft_consensus.cc:729] T d1464bf2ac73469e9a9b18135bf4a311 P e8f3740271394ed181a72f1c7a0dee5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f3740271394ed181a72f1c7a0dee5a, State: Initialized, Role: FOLLOWER
I20260926 03:31:11.495708 1287 heartbeater.cc:503] Master 127.0.97.62:46823 was elected leader, sending a full tablet report...
I20260926 03:31:11.495982 1440 consensus_queue.cc:260] T d1464bf2ac73469e9a9b18135bf4a311 P a0d2886d3bb34a179c070a4986291e68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d2886d3bb34a179c070a4986291e68" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33583 } } peers { permanent_uuid: "e8f3740271394ed181a72f1c7a0dee5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42983 } } peers { permanent_uuid: "a5c133bdea074d15a00d030f58842177" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38023 } }
I20260926 03:31:11.496488 1441 ts_tablet_manager.cc:1428] T d1464bf2ac73469e9a9b18135bf4a311 P a5c133bdea074d15a00d030f58842177: Time spent starting tablet: real 0.015s user 0.007s sys 0.007s
I20260926 03:31:11.497258 1421 heartbeater.cc:503] Master 127.0.97.62:46823 was elected leader, sending a full tablet report...
I20260926 03:31:11.498003 1440 ts_tablet_manager.cc:1428] T d1464bf2ac73469e9a9b18135bf4a311 P a0d2886d3bb34a179c070a4986291e68: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
I20260926 03:31:11.498245 1442 consensus_queue.cc:260] T d1464bf2ac73469e9a9b18135bf4a311 P e8f3740271394ed181a72f1c7a0dee5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d2886d3bb34a179c070a4986291e68" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33583 } } peers { permanent_uuid: "e8f3740271394ed181a72f1c7a0dee5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42983 } } peers { permanent_uuid: "a5c133bdea074d15a00d030f58842177" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38023 } }
I20260926 03:31:11.499898 1442 ts_tablet_manager.cc:1428] T d1464bf2ac73469e9a9b18135bf4a311 P e8f3740271394ed181a72f1c7a0dee5a: Time spent starting tablet: real 0.018s user 0.011s sys 0.005s
W20260926 03:31:11.505751 1288 tablet.cc:2369] T d1464bf2ac73469e9a9b18135bf4a311 P a5c133bdea074d15a00d030f58842177: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:31:11.508445 1447 raft_consensus.cc:482] T d1464bf2ac73469e9a9b18135bf4a311 P a0d2886d3bb34a179c070a4986291e68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:31:11.508751 1447 raft_consensus.cc:504] T d1464bf2ac73469e9a9b18135bf4a311 P a0d2886d3bb34a179c070a4986291e68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d2886d3bb34a179c070a4986291e68" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33583 } } peers { permanent_uuid: "e8f3740271394ed181a72f1c7a0dee5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42983 } } peers { permanent_uuid: "a5c133bdea074d15a00d030f58842177" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38023 } }
I20260926 03:31:11.510460 1447 leader_election.cc:288] T d1464bf2ac73469e9a9b18135bf4a311 P a0d2886d3bb34a179c070a4986291e68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8f3740271394ed181a72f1c7a0dee5a (127.0.97.1:42983), a5c133bdea074d15a00d030f58842177 (127.0.97.2:38023)
I20260926 03:31:11.517455 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:11.511049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46443 (local address 127.0.97.1:42983)
0926 03:31:11.511319 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:11.511327 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:11.511348 (+ 21us) server_negotiation.cc:407] Connection header received
0926 03:31:11.511422 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:11.511426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:11.511522 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:11.511615 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:31:11.512114 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:11.512970 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:11.515659 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:11.516486 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:11.516670 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:11.516780 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:11.517035 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 03:31:11.517171 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 03:31:11.517442 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:11.510871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42983 (local address 127.0.97.3:46443)
0926 03:31:11.511073 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:11.511112 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:11.511250 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:11.511755 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:11.511762 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:11.511793 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:31:11.511973 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:11.511981 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:11.513654 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:11.513658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:11.515498 (+ 1840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:11.515510 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:11.516677 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:11.516683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:11.516876 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:11.516902 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 03:31:11.517011 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 03:31:11.517149 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 03:31:11.518462 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:11.511412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44091 (local address 127.0.97.2:38023)
0926 03:31:11.511643 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:11.511649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:11.512112 (+ 463us) server_negotiation.cc:407] Connection header received
0926 03:31:11.512316 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:11.512320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:11.512381 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:11.512471 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:31:11.513828 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:11.514941 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:11.516623 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:11.517451 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:11.517615 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:11.517733 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:11.518059 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 03:31:11.518199 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:31:11.518498 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:11.511323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38023 (local address 127.0.97.3:44091)
0926 03:31:11.512007 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:11.512022 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:11.512173 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:11.512497 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:11.512501 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:11.512525 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:31:11.513652 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:11.513665 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:11.515105 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:11.515109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:11.516433 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:11.516444 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:11.517778 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:11.517786 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:11.517903 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:11.517926 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 03:31:11.518033 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 03:31:11.518162 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":345,"threads_started":1}
I20260926 03:31:11.518872 1108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1464bf2ac73469e9a9b18135bf4a311" candidate_uuid: "a0d2886d3bb34a179c070a4986291e68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8f3740271394ed181a72f1c7a0dee5a" is_pre_election: true
I20260926 03:31:11.519559 1108 raft_consensus.cc:2454] T d1464bf2ac73469e9a9b18135bf4a311 P e8f3740271394ed181a72f1c7a0dee5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d2886d3bb34a179c070a4986291e68 in term 0.
I20260926 03:31:11.519601 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1464bf2ac73469e9a9b18135bf4a311" candidate_uuid: "a0d2886d3bb34a179c070a4986291e68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5c133bdea074d15a00d030f58842177" is_pre_election: true
I20260926 03:31:11.520208 1242 raft_consensus.cc:2454] T d1464bf2ac73469e9a9b18135bf4a311 P a5c133bdea074d15a00d030f58842177 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d2886d3bb34a179c070a4986291e68 in term 0.
I20260926 03:31:11.520421 1310 leader_election.cc:302] T d1464bf2ac73469e9a9b18135bf4a311 P a0d2886d3bb34a179c070a4986291e68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d2886d3bb34a179c070a4986291e68, e8f3740271394ed181a72f1c7a0dee5a; no voters:
I20260926 03:31:11.520900 1447 raft_consensus.cc:2781] T d1464bf2ac73469e9a9b18135bf4a311 P a0d2886d3bb34a179c070a4986291e68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:31:11.521062 1447 raft_consensus.cc:482] T d1464bf2ac73469e9a9b18135bf4a311 P a0d2886d3bb34a179c070a4986291e68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:31:11.521184 1447 raft_consensus.cc:3037] T d1464bf2ac73469e9a9b18135bf4a311 P a0d2886d3bb34a179c070a4986291e68 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:11.523722 1447 raft_consensus.cc:504] T d1464bf2ac73469e9a9b18135bf4a311 P a0d2886d3bb34a179c070a4986291e68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d2886d3bb34a179c070a4986291e68" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33583 } } peers { permanent_uuid: "e8f3740271394ed181a72f1c7a0dee5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42983 } } peers { permanent_uuid: "a5c133bdea074d15a00d030f58842177" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38023 } }
I20260926 03:31:11.524341 1447 leader_election.cc:288] T d1464bf2ac73469e9a9b18135bf4a311 P a0d2886d3bb34a179c070a4986291e68 [CANDIDATE]: Term 1 election: Requested vote from peers e8f3740271394ed181a72f1c7a0dee5a (127.0.97.1:42983), a5c133bdea074d15a00d030f58842177 (127.0.97.2:38023)
I20260926 03:31:11.525070 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1464bf2ac73469e9a9b18135bf4a311" candidate_uuid: "a0d2886d3bb34a179c070a4986291e68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5c133bdea074d15a00d030f58842177"
I20260926 03:31:11.525072 1108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1464bf2ac73469e9a9b18135bf4a311" candidate_uuid: "a0d2886d3bb34a179c070a4986291e68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8f3740271394ed181a72f1c7a0dee5a"
I20260926 03:31:11.525323 1242 raft_consensus.cc:3037] T d1464bf2ac73469e9a9b18135bf4a311 P a5c133bdea074d15a00d030f58842177 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:11.525323 1108 raft_consensus.cc:3037] T d1464bf2ac73469e9a9b18135bf4a311 P e8f3740271394ed181a72f1c7a0dee5a [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:11.529937 1108 raft_consensus.cc:2454] T d1464bf2ac73469e9a9b18135bf4a311 P e8f3740271394ed181a72f1c7a0dee5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d2886d3bb34a179c070a4986291e68 in term 1.
I20260926 03:31:11.529934 1242 raft_consensus.cc:2454] T d1464bf2ac73469e9a9b18135bf4a311 P a5c133bdea074d15a00d030f58842177 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d2886d3bb34a179c070a4986291e68 in term 1.
I20260926 03:31:11.530375 1310 leader_election.cc:302] T d1464bf2ac73469e9a9b18135bf4a311 P a0d2886d3bb34a179c070a4986291e68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d2886d3bb34a179c070a4986291e68, e8f3740271394ed181a72f1c7a0dee5a; no voters:
I20260926 03:31:11.530674 1447 raft_consensus.cc:2781] T d1464bf2ac73469e9a9b18135bf4a311 P a0d2886d3bb34a179c070a4986291e68 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:31:11.531164 1447 raft_consensus.cc:686] T d1464bf2ac73469e9a9b18135bf4a311 P a0d2886d3bb34a179c070a4986291e68 [term 1 LEADER]: Becoming Leader. State: Replica: a0d2886d3bb34a179c070a4986291e68, State: Running, Role: LEADER
I20260926 03:31:11.531735 1447 consensus_queue.cc:237] T d1464bf2ac73469e9a9b18135bf4a311 P a0d2886d3bb34a179c070a4986291e68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d2886d3bb34a179c070a4986291e68" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33583 } } peers { permanent_uuid: "e8f3740271394ed181a72f1c7a0dee5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42983 } } peers { permanent_uuid: "a5c133bdea074d15a00d030f58842177" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38023 } }
I20260926 03:31:11.537774 962 catalog_manager.cc:5495] T d1464bf2ac73469e9a9b18135bf4a311 P a0d2886d3bb34a179c070a4986291e68 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d2886d3bb34a179c070a4986291e68 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "a0d2886d3bb34a179c070a4986291e68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d2886d3bb34a179c070a4986291e68" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8f3740271394ed181a72f1c7a0dee5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5c133bdea074d15a00d030f58842177" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38023 } health_report { overall_health: UNKNOWN } } }
W20260926 03:31:11.570319 1422 tablet.cc:2369] T d1464bf2ac73469e9a9b18135bf4a311 P a0d2886d3bb34a179c070a4986291e68: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 03:31:11 dist-test-slave-dt7p krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:31:11.591094 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:11.572587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44286 (local address 127.0.97.1:42983)
0926 03:31:11.572821 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:11.572832 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:11.573424 (+ 592us) server_negotiation.cc:407] Connection header received
0926 03:31:11.573622 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:11.573627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:11.573712 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:11.573813 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:11.576315 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:11.577369 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:11.579115 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:11.579861 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:11.580009 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:11.584357 (+ 4348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:11.584374 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:11.584394 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:11.584431 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:11.587818 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:11.588792 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:11.588804 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:11.588807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:11.588987 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:11.589381 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:11.589386 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:11.589389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:11.589847 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:11.590182 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:11.590464 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 03:31:11.590712 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 03:31:11 dist-test-slave-dt7p krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393471, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 03:31:11 dist-test-slave-dt7p krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393471, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:31:11.644209 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:11.628992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44296 (local address 127.0.97.1:42983)
0926 03:31:11.629234 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:11.629244 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:11.629637 (+ 393us) server_negotiation.cc:407] Connection header received
0926 03:31:11.629889 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:11.629895 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:11.629973 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:11.630086 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:11.632254 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:11.633129 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:11.634406 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:11.634860 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:11.634990 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:11.639003 (+ 4013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:11.639008 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:11.639012 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:11.639049 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:11.641586 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:11.642206 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:11.642217 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:11.642220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:11.642358 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:11.642758 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:11.642765 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:11.642768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:11.643053 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:11.643345 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:11.643606 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 03:31:11.643853 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 03:31:11.647853 1154 tablet.cc:2369] T d1464bf2ac73469e9a9b18135bf4a311 P e8f3740271394ed181a72f1c7a0dee5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 03:31:11 dist-test-slave-dt7p krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 03:31:11 dist-test-slave-dt7p krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:31:11.695112 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:11.679132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44310 (local address 127.0.97.1:42983)
0926 03:31:11.679428 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:11.679436 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:11.679588 (+ 152us) server_negotiation.cc:407] Connection header received
0926 03:31:11.679839 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:11.679844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:11.679916 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:11.680034 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:11.681662 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:11.682426 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:11.683701 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:11.684145 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:11.684291 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:11.689713 (+ 5422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:11.689719 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:11.689722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:11.689761 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:11.692425 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:11.693152 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:11.693156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:11.693159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:11.693279 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:11.693823 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:11.693826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:11.693828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:11.694121 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:11.694329 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:11.694561 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 03:31:11.694754 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
W20260926 03:31:11.697429 1068 tablet_service.cc:2908] Rejecting scan request for tablet d1464bf2ac73469e9a9b18135bf4a311: Uninitialized: clean time has not yet been initialized
I20260926 03:31:11.701803 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 1025
I20260926 03:31:11.714697 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:11.702751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44326 (local address 127.0.97.1:42983)
0926 03:31:11.702914 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:11.702919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:11.703370 (+ 451us) server_negotiation.cc:407] Connection header received
0926 03:31:11.703511 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:11.703515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:11.703574 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:11.703653 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:11.704940 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:11.705683 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:11.706924 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:11.707360 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:11.707472 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:11.709682 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:11.709687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:11.709690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:11.709726 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:11.712423 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:11.713084 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:11.713088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:11.713090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:11.713204 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:11.713543 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:11.713547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:11.713549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:11.713832 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:11.714066 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:11.714272 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 03:31:11.714441 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 03:31:11.715389 1148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:31:11.812744 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 1025
I20260926 03:31:11.832957 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 1159
Sep 26 03:31:11 dist-test-slave-dt7p krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:31:11.849529 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:11.834086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49674 (local address 127.0.97.2:38023)
0926 03:31:11.834278 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:11.834304 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:11.834325 (+ 21us) server_negotiation.cc:407] Connection header received
0926 03:31:11.834515 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:11.834520 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:11.834603 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:11.834703 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:11.835961 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:11.836877 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:11.838741 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:11.839158 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:11.839290 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:11.843457 (+ 4167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:11.843475 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:11.843500 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:11.843535 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:11.846928 (+ 3393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:11.847579 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:11.847587 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:11.847590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:11.847716 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:11.848026 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:11.848029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:11.848031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:11.848438 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:11.848768 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:11.848936 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 03:31:11.849107 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:31:11.850309 1282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:31:11.956184 1447 consensus_queue.cc:1035] T d1464bf2ac73469e9a9b18135bf4a311 P a0d2886d3bb34a179c070a4986291e68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5c133bdea074d15a00d030f58842177" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:31:11.956867 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 1159
W20260926 03:31:11.981717 1310 connection.cc:535] client connection to 127.0.97.2:38023 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 03:31:11.982075 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 1293
W20260926 03:31:11.982131 1310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 03:31:11.984462 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:11.983836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38023 (local address 127.0.97.3:47699)
0926 03:31:11.984101 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:11.984248 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.2:38023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
W20260926 03:31:11.985020 1310 consensus_peers.cc:487] T d1464bf2ac73469e9a9b18135bf4a311 P a0d2886d3bb34a179c070a4986291e68 -> Peer a5c133bdea074d15a00d030f58842177 (127.0.97.2:38023): Couldn't send request to peer a5c133bdea074d15a00d030f58842177. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.2:38023: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
Sep 26 03:31:11 dist-test-slave-dt7p krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:31:12.002950 1466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:11.983320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56536 (local address 127.0.97.3:33583)
0926 03:31:11.984153 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:11.984159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:11.984178 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:31:11.984273 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:11.984278 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:11.984361 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:11.984498 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:11.986624 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:11.987643 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:11.990059 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:11.990809 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:11.990956 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:11.996880 (+ 5924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:11.996902 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:11.996927 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:11.996960 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:11.999951 (+ 2991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:12.000898 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:12.000906 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:12.000909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:12.001038 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:12.001408 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:12.001411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:12.001413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:12.001811 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:12.002075 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:12.002361 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 03:31:12.002563 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":1351,"threads_started":1}
I20260926 03:31:12.003675 1416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:31:12.098150 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 1293
I20260926 03:31:12.116202 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 932
I20260926 03:31:12.117708 994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:31:12.240489 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 932
2026-09-26T03:31:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4596 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:31:12 dist-test-slave-dt7p krb5kdc[1471](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:31:12 dist-test-slave-dt7p krb5kdc[1471](info): set up 2 sockets
Sep 26 03:31:12 dist-test-slave-dt7p krb5kdc[1471](info): commencing operation
krb5kdc: starting...
W20260926 03:31:13.377010 388 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:31:13 dist-test-slave-dt7p krb5kdc[1471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:31:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:31:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:31:13.554473 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41867
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:13.902374 1484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:13.902665 1484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:13.902757 1484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:13.912294 1484 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:31:13.912405 1484 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 03:31:13.912446 1484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:13.912508 1484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:13.912604 1484 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:31:13.912698 1484 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:31:13.922901 1484 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41867
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:13.924261 1484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:13.925776 1484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:13.937964 1489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:13.938087 1492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:13.938067 1490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:13.940245 1484 server_base.cc:1034] running on GCE node
I20260926 03:31:13.941098 1484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:13.942567 1484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:13.943785 1484 hybrid_clock.cc:648] HybridClock initialized: now 1790393473943729 us; error 30 us; skew 500 ppm
I20260926 03:31:13.944088 1484 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:13 dist-test-slave-dt7p krb5kdc[1471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:13.947327 1484 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:31:13.948913 1484 webserver.cc:460] Webserver started at http://127.0.97.62:33389/ using document root <none> and password file <none>
I20260926 03:31:13.949695 1484 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:13.949808 1484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:13.950174 1484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:13.953219 1484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "f0134055fc0949a1967e0700d4c50ab4"
format_stamp: "Formatted at 2026-09-26 03:31:13 on dist-test-slave-dt7p"
server_key: "bdefeba05c46d136bbc9f225567b2647"
server_key_iv: "90de37f6da1e45c773a0ff6d96f17302"
server_key_version: "encryptionkey@0"
I20260926 03:31:13.954262 1484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "f0134055fc0949a1967e0700d4c50ab4"
format_stamp: "Formatted at 2026-09-26 03:31:13 on dist-test-slave-dt7p"
server_key: "bdefeba05c46d136bbc9f225567b2647"
server_key_iv: "90de37f6da1e45c773a0ff6d96f17302"
server_key_version: "encryptionkey@0"
I20260926 03:31:13.959753 1484 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:31:13.963354 1499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:13.964761 1484 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:31:13.964936 1484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "f0134055fc0949a1967e0700d4c50ab4"
format_stamp: "Formatted at 2026-09-26 03:31:13 on dist-test-slave-dt7p"
server_key: "bdefeba05c46d136bbc9f225567b2647"
server_key_iv: "90de37f6da1e45c773a0ff6d96f17302"
server_key_version: "encryptionkey@0"
I20260926 03:31:13.965160 1484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:13.978499 1484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:13.982932 1484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:13.983228 1484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:14.005650 1484 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41867
I20260926 03:31:14.005668 1560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41867 every 8 connection(s)
I20260926 03:31:14.007166 1484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:31:14.011360 1561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:14.011686 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 1484
I20260926 03:31:14.011900 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:31:14.012651 388 external_mini_cluster.cc:1428] Setting key 97c5c18a766cfb1c91e3d80f7c510c6d
I20260926 03:31:14.024712 1561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0134055fc0949a1967e0700d4c50ab4: Bootstrap starting.
Sep 26 03:31:14 dist-test-slave-dt7p krb5kdc[1471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:31:14.028324 1561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0134055fc0949a1967e0700d4c50ab4: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:14.029695 1561 log.cc:824] T 00000000000000000000000000000000 P f0134055fc0949a1967e0700d4c50ab4: Log is configured to *not* fsync() on all Append() calls
I20260926 03:31:14.033107 1561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0134055fc0949a1967e0700d4c50ab4: No bootstrap required, opened a new log
I20260926 03:31:14.036420 1564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:14.014771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60140 (local address 127.0.97.62:41867)
0926 03:31:14.015658 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:14.015670 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:14.015717 (+ 47us) server_negotiation.cc:407] Connection header received
0926 03:31:14.017674 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:14.017701 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:14.018158 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:14.018619 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:14.020149 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:14.021590 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:14.022877 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:14.023594 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:14.023794 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:14.028297 (+ 4503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:14.028306 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:14.028326 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:14.028365 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:14.032048 (+ 3683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:14.032911 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:14.032918 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:14.032925 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:14.033108 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:14.033495 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:14.033502 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:14.033506 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:14.034076 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:14.034378 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:14.034866 (+ 488us) server_negotiation.cc:299] Negotiation successful
0926 03:31:14.035312 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":382,"threads_started":1}
I20260926 03:31:14.041715 1561 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0134055fc0949a1967e0700d4c50ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0134055fc0949a1967e0700d4c50ab4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41867 } }
I20260926 03:31:14.042131 1561 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0134055fc0949a1967e0700d4c50ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:14.042245 1561 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0134055fc0949a1967e0700d4c50ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0134055fc0949a1967e0700d4c50ab4, State: Initialized, Role: FOLLOWER
I20260926 03:31:14.042886 1561 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0134055fc0949a1967e0700d4c50ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0134055fc0949a1967e0700d4c50ab4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41867 } }
I20260926 03:31:14.043133 1561 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0134055fc0949a1967e0700d4c50ab4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:31:14.043278 1561 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0134055fc0949a1967e0700d4c50ab4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:31:14.043463 1561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0134055fc0949a1967e0700d4c50ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:14.045907 1561 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0134055fc0949a1967e0700d4c50ab4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0134055fc0949a1967e0700d4c50ab4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41867 } }
I20260926 03:31:14.046391 1561 leader_election.cc:302] T 00000000000000000000000000000000 P f0134055fc0949a1967e0700d4c50ab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0134055fc0949a1967e0700d4c50ab4; no voters:
I20260926 03:31:14.046931 1561 leader_election.cc:288] T 00000000000000000000000000000000 P f0134055fc0949a1967e0700d4c50ab4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:31:14.047115 1566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0134055fc0949a1967e0700d4c50ab4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:31:14.048144 1566 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0134055fc0949a1967e0700d4c50ab4 [term 1 LEADER]: Becoming Leader. State: Replica: f0134055fc0949a1967e0700d4c50ab4, State: Running, Role: LEADER
I20260926 03:31:14.048774 1566 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0134055fc0949a1967e0700d4c50ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0134055fc0949a1967e0700d4c50ab4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41867 } }
I20260926 03:31:14.049306 1561 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0134055fc0949a1967e0700d4c50ab4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:31:14.051571 1568 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0134055fc0949a1967e0700d4c50ab4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0134055fc0949a1967e0700d4c50ab4. Latest consensus state: current_term: 1 leader_uuid: "f0134055fc0949a1967e0700d4c50ab4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0134055fc0949a1967e0700d4c50ab4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41867 } } }
I20260926 03:31:14.052095 1567 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0134055fc0949a1967e0700d4c50ab4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0134055fc0949a1967e0700d4c50ab4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0134055fc0949a1967e0700d4c50ab4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41867 } } }
I20260926 03:31:14.052774 1568 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0134055fc0949a1967e0700d4c50ab4 [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:14.053078 1567 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0134055fc0949a1967e0700d4c50ab4 [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:14.054623 1576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:31:14.059928 1576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:31:14.068437 1576 catalog_manager.cc:1347] Generated new cluster ID: e6fd58925a614a4d96bbc2d382d06671
I20260926 03:31:14.068591 1576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:31:14.082639 1576 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:31:14.084041 1576 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:31:14.095599 1576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0134055fc0949a1967e0700d4c50ab4: Generated new TSK 0
I20260926 03:31:14.096488 1576 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:31:14.144081 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41867
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:14.476460 1587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:14.476781 1587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:14.476881 1587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:14.484113 1587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:14.484225 1587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:14.484306 1587 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:31:14.484386 1587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:31:14.494251 1587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:41867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:14.495466 1587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:14.496898 1587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:14.509900 1592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:14.509781 1595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:14.509814 1593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:14.510829 1587 server_base.cc:1034] running on GCE node
I20260926 03:31:14.511435 1587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:14.512353 1587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:14.513612 1587 hybrid_clock.cc:648] HybridClock initialized: now 1790393474513532 us; error 54 us; skew 500 ppm
I20260926 03:31:14.513913 1587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:14 dist-test-slave-dt7p krb5kdc[1471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:14.517036 1587 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:31:14.518438 1587 webserver.cc:460] Webserver started at http://127.0.97.1:35741/ using document root <none> and password file <none>
I20260926 03:31:14.519147 1587 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:14.519279 1587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:14.519728 1587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:14.523025 1587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-0/data/instance:
uuid: "4520160883e74babb2b82b6c74c5f265"
format_stamp: "Formatted at 2026-09-26 03:31:14 on dist-test-slave-dt7p"
server_key: "36ea64fcd929645816666ed1e5524ecc"
server_key_iv: "694af6d2502c322687cc8b5cea83994d"
server_key_version: "encryptionkey@0"
I20260926 03:31:14.523833 1587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4520160883e74babb2b82b6c74c5f265"
format_stamp: "Formatted at 2026-09-26 03:31:14 on dist-test-slave-dt7p"
server_key: "36ea64fcd929645816666ed1e5524ecc"
server_key_iv: "694af6d2502c322687cc8b5cea83994d"
server_key_version: "encryptionkey@0"
I20260926 03:31:14.530072 1587 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.008s
I20260926 03:31:14.534133 1602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:14.535599 1587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20260926 03:31:14.535789 1587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-0/wal
uuid: "4520160883e74babb2b82b6c74c5f265"
format_stamp: "Formatted at 2026-09-26 03:31:14 on dist-test-slave-dt7p"
server_key: "36ea64fcd929645816666ed1e5524ecc"
server_key_iv: "694af6d2502c322687cc8b5cea83994d"
server_key_version: "encryptionkey@0"
I20260926 03:31:14.536067 1587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:14.557149 1587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:14.561991 1587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:14.562278 1587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:14.565122 1587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:31:14.565227 1587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:14.565361 1587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:31:14.565439 1587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:31:14 dist-test-slave-dt7p krb5kdc[1471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:31:14.595206 1612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:14.566575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41867 (local address 127.0.97.1:32995)
0926 03:31:14.567426 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:14.567475 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:14.570000 (+ 2525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:14.570967 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:14.570982 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:14.572004 (+ 1022us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:14.573283 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:14.573320 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:14.574762 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:14.574770 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:14.577758 (+ 2988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:14.577779 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:14.578900 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:14.578910 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:14.579279 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:14.580178 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:14.580219 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:14.583661 (+ 3442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:14.588167 (+ 4506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:14.588179 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:14.588204 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:14.589039 (+ 835us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:14.589585 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:14.589593 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:14.589599 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:14.589997 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:14.590827 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:14.590842 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:14.591464 (+ 622us) client_negotiation.cc:769] Sending connection context
0926 03:31:14.591843 (+ 379us) client_negotiation.cc:240] Negotiation successful
0926 03:31:14.592648 (+ 805us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":391,"threads_started":1}
I20260926 03:31:14.596246 1611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:14.565896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:32995 (local address 127.0.97.62:41867)
0926 03:31:14.566420 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:14.566425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:14.567620 (+ 1195us) server_negotiation.cc:407] Connection header received
0926 03:31:14.570375 (+ 2755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:14.570380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:14.570482 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:14.570606 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:14.573558 (+ 2952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:14.574579 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:14.578003 (+ 3424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:14.578593 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:14.578756 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:14.583952 (+ 5196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:14.583962 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:14.583970 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:14.584027 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:14.587962 (+ 3935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:14.589253 (+ 1291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:14.589262 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:14.589269 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:14.589436 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:14.590174 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:14.590181 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:14.590186 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:14.590655 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:14.594531 (+ 3876us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:14.594696 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 03:31:14.594947 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":271,"threads_started":1}
I20260926 03:31:14.617797 1611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:14.602462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37893 (local address 127.0.97.62:41867)
0926 03:31:14.602669 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:14.602677 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:14.603333 (+ 656us) server_negotiation.cc:407] Connection header received
0926 03:31:14.603533 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:14.603536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:14.603604 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:14.603683 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:14.605338 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:14.606256 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:14.608584 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:14.609091 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:14.609236 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:14.611809 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:14.611814 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:14.611817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:14.611843 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:14.614805 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:14.615624 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:14.615631 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:14.615637 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:14.615867 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:14.616340 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:14.616343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:14.616345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:14.616665 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:14.616814 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:14.617213 (+ 399us) server_negotiation.cc:299] Negotiation successful
0926 03:31:14.617419 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 03:31:14.617893 1693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:14.602461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41867 (local address 127.0.97.1:37893)
0926 03:31:14.603214 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:14.603239 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:14.603404 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:14.603747 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:14.603754 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:14.604504 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:14.605137 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:14.605159 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:14.606431 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:14.606440 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:14.608399 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:14.608432 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:14.609292 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:14.609300 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:14.609571 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:14.610256 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:14.610285 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:14.611628 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:14.614942 (+ 3314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:14.614947 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:14.614951 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:14.615486 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:14.616008 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:14.616020 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:14.616026 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:14.616219 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:14.616782 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:14.616787 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:14.616965 (+ 178us) client_negotiation.cc:769] Sending connection context
0926 03:31:14.617131 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 03:31:14.617438 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":390,"threads_started":1}
I20260926 03:31:14.621096 1587 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42837
I20260926 03:31:14.621131 1722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42837 every 8 connection(s)
I20260926 03:31:14.622478 1587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:31:14.622879 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 1587
I20260926 03:31:14.623121 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:31:14.623916 388 external_mini_cluster.cc:1428] Setting key 1cc04ed6f3034e723c4c44fbcf7864e6
I20260926 03:31:14.624836 1723 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41867
I20260926 03:31:14.625146 1723 heartbeater.cc:464] Registering TS with master...
I20260926 03:31:14.625957 1723 heartbeater.cc:511] Master 127.0.97.62:41867 requested a full tablet report, sending...
I20260926 03:31:14.627956 1516 ts_manager.cc:194] Registered new tserver with Master: 4520160883e74babb2b82b6c74c5f265 (127.0.97.1:42837)
I20260926 03:31:14.629463 1516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:32995
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:31:14.661896 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41867
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:14.981622 1727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:14.981916 1727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:14.982014 1727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:14.989312 1727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:14.989421 1727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:14.989502 1727 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:31:14.989591 1727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:31:14.999092 1727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:41867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:15.000353 1727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:15.001662 1727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:15.015399 1733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:15.015504 1735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:15.015420 1732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:15.016551 1727 server_base.cc:1034] running on GCE node
I20260926 03:31:15.017278 1727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:15.018345 1727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:15.019601 1727 hybrid_clock.cc:648] HybridClock initialized: now 1790393475019519 us; error 60 us; skew 500 ppm
I20260926 03:31:15.019961 1727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:15 dist-test-slave-dt7p krb5kdc[1471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:15.023692 1727 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:31:15.025269 1727 webserver.cc:460] Webserver started at http://127.0.97.2:33987/ using document root <none> and password file <none>
I20260926 03:31:15.025988 1727 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:15.026122 1727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:15.026456 1727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:15.029495 1727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-1/data/instance:
uuid: "171b9f87cc9d4281838397ec8c90d583"
format_stamp: "Formatted at 2026-09-26 03:31:15 on dist-test-slave-dt7p"
server_key: "dce98e917afe9fa23454e7b8fd7c135d"
server_key_iv: "bced94250a26c4b5737f324120ff9bdd"
server_key_version: "encryptionkey@0"
I20260926 03:31:15.030279 1727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-1/wal/instance:
uuid: "171b9f87cc9d4281838397ec8c90d583"
format_stamp: "Formatted at 2026-09-26 03:31:15 on dist-test-slave-dt7p"
server_key: "dce98e917afe9fa23454e7b8fd7c135d"
server_key_iv: "bced94250a26c4b5737f324120ff9bdd"
server_key_version: "encryptionkey@0"
I20260926 03:31:15.035578 1727 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:31:15.039371 1742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:15.040969 1727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:31:15.041106 1727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-1/wal
uuid: "171b9f87cc9d4281838397ec8c90d583"
format_stamp: "Formatted at 2026-09-26 03:31:15 on dist-test-slave-dt7p"
server_key: "dce98e917afe9fa23454e7b8fd7c135d"
server_key_iv: "bced94250a26c4b5737f324120ff9bdd"
server_key_version: "encryptionkey@0"
I20260926 03:31:15.041294 1727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:15.065676 1727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:15.070339 1727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:15.070585 1727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:15.073143 1727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:31:15.073264 1727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:15.073418 1727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:31:15.073485 1727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:31:15 dist-test-slave-dt7p krb5kdc[1471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:31:15.099215 1611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:15.074031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46289 (local address 127.0.97.62:41867)
0926 03:31:15.074227 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:15.074233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:15.075782 (+ 1549us) server_negotiation.cc:407] Connection header received
0926 03:31:15.078999 (+ 3217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:15.079005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:15.079119 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:15.079241 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:15.081411 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.082513 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.085083 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.085712 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.085844 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:15.091388 (+ 5544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:15.091393 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:15.091396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:15.091432 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:15.094999 (+ 3567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:15.096066 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:15.096075 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:15.096080 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:15.096220 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:15.096818 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:15.096826 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:15.096832 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:15.097302 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:15.097492 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:15.098336 (+ 844us) server_negotiation.cc:299] Negotiation successful
0926 03:31:15.098654 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 03:31:15.100077 1751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:15.074658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41867 (local address 127.0.97.2:46289)
0926 03:31:15.075597 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:15.075643 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:15.078567 (+ 2924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:15.079494 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:15.079502 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:15.080245 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:15.081214 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:15.081248 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.082743 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.082751 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:15.084849 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:15.084865 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.085972 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.085984 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:15.086322 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:15.087197 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:15.087238 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:15.091090 (+ 3852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:15.095162 (+ 4072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:15.095174 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:15.095199 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:15.095888 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:15.096341 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:15.096350 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:15.096354 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:15.096642 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:15.097470 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:15.097481 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:15.097907 (+ 426us) client_negotiation.cc:769] Sending connection context
0926 03:31:15.098144 (+ 237us) client_negotiation.cc:240] Negotiation successful
0926 03:31:15.098715 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":386,"threads_started":1}
I20260926 03:31:15.119519 1727 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36863
I20260926 03:31:15.119539 1861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36863 every 8 connection(s)
I20260926 03:31:15.120894 1727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:31:15.121290 1611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:15.106702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57151 (local address 127.0.97.62:41867)
0926 03:31:15.106930 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:15.106939 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:15.107752 (+ 813us) server_negotiation.cc:407] Connection header received
0926 03:31:15.108056 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:15.108062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:15.108136 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:15.108236 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:15.109606 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.110510 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.112424 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.113080 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.113195 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:15.115616 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:15.115622 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:15.115626 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:15.115667 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:15.118529 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:15.119358 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:15.119364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:15.119368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:15.119501 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:15.119858 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:15.119862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:15.119864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:15.120182 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:15.120331 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:15.120748 (+ 417us) server_negotiation.cc:299] Negotiation successful
0926 03:31:15.120958 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 03:31:15.121449 1829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:15.106805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41867 (local address 127.0.97.2:57151)
0926 03:31:15.107624 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:15.107653 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:15.107881 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:15.108263 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:15.108267 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:15.108876 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:15.109471 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:15.109485 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.110697 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.110701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:15.112283 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:15.112294 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.113308 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.113320 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:15.113477 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:15.114211 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:15.114228 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:15.115435 (+ 1207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:15.118662 (+ 3227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:15.118667 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:15.118672 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:15.119231 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:15.119611 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:15.119616 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:15.119620 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:15.119747 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:15.120299 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:15.120304 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:15.120486 (+ 182us) client_negotiation.cc:769] Sending connection context
0926 03:31:15.120654 (+ 168us) client_negotiation.cc:240] Negotiation successful
0926 03:31:15.120988 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":501,"threads_started":1}
I20260926 03:31:15.124491 1862 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41867
I20260926 03:31:15.124740 1862 heartbeater.cc:464] Registering TS with master...
I20260926 03:31:15.125579 1862 heartbeater.cc:511] Master 127.0.97.62:41867 requested a full tablet report, sending...
I20260926 03:31:15.126859 1516 ts_manager.cc:194] Registered new tserver with Master: 171b9f87cc9d4281838397ec8c90d583 (127.0.97.2:36863)
I20260926 03:31:15.127610 1516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:46289
I20260926 03:31:15.129307 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 1727
I20260926 03:31:15.129510 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:31:15.130038 388 external_mini_cluster.cc:1428] Setting key f6c3a4bb50d4b5881e7ecd92d7563977
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:31:15.171806 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41867
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:15.495656 1866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:15.495958 1866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:15.496181 1866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:15.504581 1866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:15.504702 1866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:15.504894 1866 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:31:15.505041 1866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:31:15.515738 1866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:41867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:15.517066 1866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:15.518393 1866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:15.531576 1874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:15.531610 1872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:15.531646 1871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:15.532266 1866 server_base.cc:1034] running on GCE node
I20260926 03:31:15.532994 1866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:15.533926 1866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:15.535182 1866 hybrid_clock.cc:648] HybridClock initialized: now 1790393475535104 us; error 53 us; skew 500 ppm
I20260926 03:31:15.535491 1866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:15 dist-test-slave-dt7p krb5kdc[1471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:15.538589 1866 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:31:15.540081 1866 webserver.cc:460] Webserver started at http://127.0.97.3:38197/ using document root <none> and password file <none>
I20260926 03:31:15.540889 1866 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:15.541021 1866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:15.541365 1866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:15.544191 1866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-2/data/instance:
uuid: "76bbe52cb54d4f14b4a8eb90182de87f"
format_stamp: "Formatted at 2026-09-26 03:31:15 on dist-test-slave-dt7p"
server_key: "253fb07b422e9a6354a368c5e817e5f2"
server_key_iv: "0a5603f9e30c36540bd84afc5588f1e7"
server_key_version: "encryptionkey@0"
I20260926 03:31:15.545040 1866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-2/wal/instance:
uuid: "76bbe52cb54d4f14b4a8eb90182de87f"
format_stamp: "Formatted at 2026-09-26 03:31:15 on dist-test-slave-dt7p"
server_key: "253fb07b422e9a6354a368c5e817e5f2"
server_key_iv: "0a5603f9e30c36540bd84afc5588f1e7"
server_key_version: "encryptionkey@0"
I20260926 03:31:15.550047 1866 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 03:31:15.553644 1881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:15.555047 1866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:31:15.555200 1866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-2/wal
uuid: "76bbe52cb54d4f14b4a8eb90182de87f"
format_stamp: "Formatted at 2026-09-26 03:31:15 on dist-test-slave-dt7p"
server_key: "253fb07b422e9a6354a368c5e817e5f2"
server_key_iv: "0a5603f9e30c36540bd84afc5588f1e7"
server_key_version: "encryptionkey@0"
I20260926 03:31:15.555425 1866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:15.574002 1866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:15.578424 1866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:15.578692 1866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:15.581362 1866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:31:15.581440 1866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:15.581591 1866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:31:15.581694 1866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:31:15 dist-test-slave-dt7p krb5kdc[1471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:31:15.607321 1890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:15.583177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41867 (local address 127.0.97.3:60287)
0926 03:31:15.583998 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:15.584034 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:15.586268 (+ 2234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:15.587206 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:15.587214 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:15.587984 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:15.588902 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:15.588925 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.589990 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.589997 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:15.592514 (+ 2517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:15.592531 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.593839 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.593847 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:15.594130 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:15.594956 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:15.594994 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:15.598128 (+ 3134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:15.601949 (+ 3821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:15.601957 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:15.601976 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:15.602710 (+ 734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:15.603167 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:15.603174 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:15.603178 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:15.603371 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:15.604074 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:15.604082 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:15.604612 (+ 530us) client_negotiation.cc:769] Sending connection context
0926 03:31:15.605005 (+ 393us) client_negotiation.cc:240] Negotiation successful
0926 03:31:15.605596 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":370,"threads_started":1}
I20260926 03:31:15.608311 1611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:15.582197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60287 (local address 127.0.97.62:41867)
0926 03:31:15.582391 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:15.582398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:15.584156 (+ 1758us) server_negotiation.cc:407] Connection header received
0926 03:31:15.586684 (+ 2528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:15.586692 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:15.586804 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:15.586935 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:15.589087 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.589824 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.592736 (+ 2912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.593434 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.593560 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:15.598354 (+ 4794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:15.598363 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:15.598368 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:15.598413 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:15.601808 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:15.602843 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:15.602850 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:15.602855 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:15.603027 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:15.603526 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:15.603533 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:15.603538 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:15.603939 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:15.604104 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:15.607910 (+ 3806us) server_negotiation.cc:299] Negotiation successful
0926 03:31:15.608083 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 03:31:15.632822 1723 heartbeater.cc:503] Master 127.0.97.62:41867 was elected leader, sending a full tablet report...
I20260926 03:31:15.635543 1611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:15.619263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48647 (local address 127.0.97.62:41867)
0926 03:31:15.619452 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:15.619459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:15.620091 (+ 632us) server_negotiation.cc:407] Connection header received
0926 03:31:15.620361 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:15.620370 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:15.620493 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:15.620658 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:15.621848 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.623270 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.625239 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.626099 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.626248 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:15.629181 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:15.629189 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:15.629197 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:15.629243 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:15.632506 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:15.633496 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:15.633501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:15.633504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:15.633660 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:15.634115 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:15.634120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:15.634123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:15.634442 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:15.634597 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:15.635122 (+ 525us) server_negotiation.cc:299] Negotiation successful
0926 03:31:15.635280 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:31:15.636129 1976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:15.619300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41867 (local address 127.0.97.3:48647)
0926 03:31:15.619986 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:15.620005 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:15.620198 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:15.620689 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:15.620694 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:15.621269 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:15.621692 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:15.621706 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.623446 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.623451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:15.625083 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:15.625096 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.626501 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.626511 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:15.626761 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:15.627599 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:15.627617 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:15.628880 (+ 1263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:15.632724 (+ 3844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:15.632732 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:15.632738 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:15.633363 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:15.633795 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:15.633802 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:15.633809 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:15.633994 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:15.634577 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:15.634584 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:15.634832 (+ 248us) client_negotiation.cc:769] Sending connection context
0926 03:31:15.635017 (+ 185us) client_negotiation.cc:240] Negotiation successful
0926 03:31:15.635474 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":389,"threads_started":1}
I20260926 03:31:15.637984 1866 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40979
I20260926 03:31:15.638128 2000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40979 every 8 connection(s)
I20260926 03:31:15.639354 1866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:31:15.641625 2001 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41867
I20260926 03:31:15.641803 2001 heartbeater.cc:464] Registering TS with master...
I20260926 03:31:15.642338 2001 heartbeater.cc:511] Master 127.0.97.62:41867 requested a full tablet report, sending...
I20260926 03:31:15.643469 1516 ts_manager.cc:194] Registered new tserver with Master: 76bbe52cb54d4f14b4a8eb90182de87f (127.0.97.3:40979)
I20260926 03:31:15.644287 1516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60287
I20260926 03:31:15.649626 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 1866
I20260926 03:31:15.649819 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:31:15.650288 388 external_mini_cluster.cc:1428] Setting key 0f159a516804b0497e8942efc23dcfd8
I20260926 03:31:15.653215 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 03:31:15 dist-test-slave-dt7p krb5kdc[1471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393475, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 03:31:15 dist-test-slave-dt7p krb5kdc[1471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393475, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:31:15.697405 1611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:15.684417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60152 (local address 127.0.97.62:41867)
0926 03:31:15.684665 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:15.684671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:15.684871 (+ 200us) server_negotiation.cc:407] Connection header received
0926 03:31:15.685045 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:15.685072 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:15.685159 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:15.685248 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:15.686569 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.687279 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.688443 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.688906 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.688998 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:15.692670 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:15.692675 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:15.692678 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:15.692710 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:15.695213 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:15.695725 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:15.695729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:15.695731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:15.695835 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:15.696135 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:15.696138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:15.696140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:15.696377 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:15.696540 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:15.696844 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 03:31:15.696999 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 03:31:15.701745 1516 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:31:15.708714 1516 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:31:15.732234 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:15.721295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36863 (local address 127.0.0.1:60508)
0926 03:31:15.723844 (+ 2549us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:15.723859 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:15.724048 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:15.724873 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:15.724876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:15.724893 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:31:15.725265 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:15.725279 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.727002 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.727009 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:15.729451 (+ 2442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:15.729464 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.731265 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.731272 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:15.731469 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:15.731549 (+ 80us) client_negotiation.cc:769] Sending connection context
0926 03:31:15.731687 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 03:31:15.731866 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2393,"mutex_wait_us":103,"thread_start_us":728,"threads_started":1}
I20260926 03:31:15.733136 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:15.720934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60508 (local address 127.0.97.2:36863)
0926 03:31:15.723216 (+ 2282us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:15.723227 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:15.723952 (+ 725us) server_negotiation.cc:407] Connection header received
0926 03:31:15.724237 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:15.724248 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:15.724489 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:15.724694 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:31:15.725447 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.726878 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.729609 (+ 2731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.731026 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.731176 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:15.731379 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:15.732746 (+ 1367us) server_negotiation.cc:299] Negotiation successful
0926 03:31:15.732872 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2083,"thread_start_us":2622,"threads_started":1}
I20260926 03:31:15.734174 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:15.720369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40979 (local address 127.0.0.1:47742)
0926 03:31:15.722392 (+ 2023us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:15.722427 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:15.722592 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:15.723675 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:15.723682 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:15.723714 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:31:15.724197 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:15.724212 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.726691 (+ 2479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.726695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:15.729450 (+ 2755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:15.729464 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.733158 (+ 3694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.733166 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:15.733550 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:15.733584 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 03:31:15.733670 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 03:31:15.733807 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1803,"mutex_wait_us":110,"thread_start_us":3028,"threads_started":1}
I20260926 03:31:15.734220 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:15.720614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47742 (local address 127.0.97.3:40979)
0926 03:31:15.722086 (+ 1472us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:15.722099 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:15.722798 (+ 699us) server_negotiation.cc:407] Connection header received
0926 03:31:15.722897 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:15.722904 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:15.723179 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:15.723366 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:31:15.724460 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.726506 (+ 2046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.731463 (+ 4957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.732937 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.733065 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:15.733288 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:15.733746 (+ 458us) server_negotiation.cc:299] Negotiation successful
0926 03:31:15.733894 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1249,"thread_start_us":1047,"threads_started":1}
I20260926 03:31:15.734252 2019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:15.723384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47492 (local address 127.0.97.1:42837)
0926 03:31:15.726471 (+ 3087us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:15.726478 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:15.727344 (+ 866us) server_negotiation.cc:407] Connection header received
0926 03:31:15.727446 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:15.727454 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:15.727659 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:15.727830 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:31:15.728550 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.729486 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.731842 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.733242 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.733366 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:15.733551 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:15.733827 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 03:31:15.733961 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2943,"thread_start_us":2820,"threads_started":1}
I20260926 03:31:15.734846 2018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:15.722917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42837 (local address 127.0.0.1:47492)
0926 03:31:15.726866 (+ 3949us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:15.726886 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:15.727203 (+ 317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:15.728038 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:15.728041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:15.728059 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:31:15.728388 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:15.728395 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.729675 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.729680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:15.731671 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:15.731685 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.733427 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.733433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:15.733611 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:15.733642 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 03:31:15.733727 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 03:31:15.734500 (+ 773us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3827,"mutex_wait_us":627,"thread_start_us":3704,"threads_started":1}
I20260926 03:31:15.737049 1792 tablet_service.cc:1467] Processing CreateTablet for tablet 2be6b563b5e94502afca651b2d803a3c (DEFAULT_TABLE table=test-table [id=62c3176758c644c08f80c85aa258db94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:31:15.738216 1792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2be6b563b5e94502afca651b2d803a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:15.737996 1653 tablet_service.cc:1467] Processing CreateTablet for tablet 2be6b563b5e94502afca651b2d803a3c (DEFAULT_TABLE table=test-table [id=62c3176758c644c08f80c85aa258db94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:31:15.737994 1931 tablet_service.cc:1467] Processing CreateTablet for tablet 2be6b563b5e94502afca651b2d803a3c (DEFAULT_TABLE table=test-table [id=62c3176758c644c08f80c85aa258db94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:31:15.739267 1653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2be6b563b5e94502afca651b2d803a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:15.739284 1931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2be6b563b5e94502afca651b2d803a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:15.750932 2020 tablet_bootstrap.cc:492] T 2be6b563b5e94502afca651b2d803a3c P 171b9f87cc9d4281838397ec8c90d583: Bootstrap starting.
I20260926 03:31:15.754108 2021 tablet_bootstrap.cc:492] T 2be6b563b5e94502afca651b2d803a3c P 4520160883e74babb2b82b6c74c5f265: Bootstrap starting.
I20260926 03:31:15.754273 2022 tablet_bootstrap.cc:492] T 2be6b563b5e94502afca651b2d803a3c P 76bbe52cb54d4f14b4a8eb90182de87f: Bootstrap starting.
I20260926 03:31:15.754801 2020 tablet_bootstrap.cc:654] T 2be6b563b5e94502afca651b2d803a3c P 171b9f87cc9d4281838397ec8c90d583: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:15.756119 2020 log.cc:824] T 2be6b563b5e94502afca651b2d803a3c P 171b9f87cc9d4281838397ec8c90d583: Log is configured to *not* fsync() on all Append() calls
I20260926 03:31:15.757032 2021 tablet_bootstrap.cc:654] T 2be6b563b5e94502afca651b2d803a3c P 4520160883e74babb2b82b6c74c5f265: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:15.757543 2022 tablet_bootstrap.cc:654] T 2be6b563b5e94502afca651b2d803a3c P 76bbe52cb54d4f14b4a8eb90182de87f: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:15.758241 2021 log.cc:824] T 2be6b563b5e94502afca651b2d803a3c P 4520160883e74babb2b82b6c74c5f265: Log is configured to *not* fsync() on all Append() calls
I20260926 03:31:15.758687 2022 log.cc:824] T 2be6b563b5e94502afca651b2d803a3c P 76bbe52cb54d4f14b4a8eb90182de87f: Log is configured to *not* fsync() on all Append() calls
I20260926 03:31:15.760475 2020 tablet_bootstrap.cc:492] T 2be6b563b5e94502afca651b2d803a3c P 171b9f87cc9d4281838397ec8c90d583: No bootstrap required, opened a new log
I20260926 03:31:15.760771 2020 ts_tablet_manager.cc:1397] T 2be6b563b5e94502afca651b2d803a3c P 171b9f87cc9d4281838397ec8c90d583: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260926 03:31:15.761193 2021 tablet_bootstrap.cc:492] T 2be6b563b5e94502afca651b2d803a3c P 4520160883e74babb2b82b6c74c5f265: No bootstrap required, opened a new log
I20260926 03:31:15.761269 2022 tablet_bootstrap.cc:492] T 2be6b563b5e94502afca651b2d803a3c P 76bbe52cb54d4f14b4a8eb90182de87f: No bootstrap required, opened a new log
I20260926 03:31:15.761444 2021 ts_tablet_manager.cc:1397] T 2be6b563b5e94502afca651b2d803a3c P 4520160883e74babb2b82b6c74c5f265: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 03:31:15.761497 2022 ts_tablet_manager.cc:1397] T 2be6b563b5e94502afca651b2d803a3c P 76bbe52cb54d4f14b4a8eb90182de87f: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 03:31:15.768117 2022 raft_consensus.cc:348] T 2be6b563b5e94502afca651b2d803a3c P 76bbe52cb54d4f14b4a8eb90182de87f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } } peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } }
I20260926 03:31:15.768499 2022 raft_consensus.cc:374] T 2be6b563b5e94502afca651b2d803a3c P 76bbe52cb54d4f14b4a8eb90182de87f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:15.768623 2022 raft_consensus.cc:729] T 2be6b563b5e94502afca651b2d803a3c P 76bbe52cb54d4f14b4a8eb90182de87f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76bbe52cb54d4f14b4a8eb90182de87f, State: Initialized, Role: FOLLOWER
I20260926 03:31:15.769292 2022 consensus_queue.cc:260] T 2be6b563b5e94502afca651b2d803a3c P 76bbe52cb54d4f14b4a8eb90182de87f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } } peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } }
I20260926 03:31:15.771406 2001 heartbeater.cc:503] Master 127.0.97.62:41867 was elected leader, sending a full tablet report...
I20260926 03:31:15.771615 2020 raft_consensus.cc:348] T 2be6b563b5e94502afca651b2d803a3c P 171b9f87cc9d4281838397ec8c90d583 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } } peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } }
I20260926 03:31:15.772044 2020 raft_consensus.cc:374] T 2be6b563b5e94502afca651b2d803a3c P 171b9f87cc9d4281838397ec8c90d583 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:15.772136 2022 ts_tablet_manager.cc:1428] T 2be6b563b5e94502afca651b2d803a3c P 76bbe52cb54d4f14b4a8eb90182de87f: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260926 03:31:15.772149 2020 raft_consensus.cc:729] T 2be6b563b5e94502afca651b2d803a3c P 171b9f87cc9d4281838397ec8c90d583 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171b9f87cc9d4281838397ec8c90d583, State: Initialized, Role: FOLLOWER
I20260926 03:31:15.772305 2021 raft_consensus.cc:348] T 2be6b563b5e94502afca651b2d803a3c P 4520160883e74babb2b82b6c74c5f265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } } peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } }
I20260926 03:31:15.772704 2021 raft_consensus.cc:374] T 2be6b563b5e94502afca651b2d803a3c P 4520160883e74babb2b82b6c74c5f265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:15.772799 2021 raft_consensus.cc:729] T 2be6b563b5e94502afca651b2d803a3c P 4520160883e74babb2b82b6c74c5f265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4520160883e74babb2b82b6c74c5f265, State: Initialized, Role: FOLLOWER
I20260926 03:31:15.772785 2020 consensus_queue.cc:260] T 2be6b563b5e94502afca651b2d803a3c P 171b9f87cc9d4281838397ec8c90d583 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } } peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } }
I20260926 03:31:15.773350 2021 consensus_queue.cc:260] T 2be6b563b5e94502afca651b2d803a3c P 4520160883e74babb2b82b6c74c5f265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } } peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } }
I20260926 03:31:15.773850 1862 heartbeater.cc:503] Master 127.0.97.62:41867 was elected leader, sending a full tablet report...
I20260926 03:31:15.774675 2020 ts_tablet_manager.cc:1428] T 2be6b563b5e94502afca651b2d803a3c P 171b9f87cc9d4281838397ec8c90d583: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s
I20260926 03:31:15.776232 2021 ts_tablet_manager.cc:1428] T 2be6b563b5e94502afca651b2d803a3c P 4520160883e74babb2b82b6c74c5f265: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260926 03:31:15.870836 2026 raft_consensus.cc:482] T 2be6b563b5e94502afca651b2d803a3c P 76bbe52cb54d4f14b4a8eb90182de87f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:31:15.871187 2026 raft_consensus.cc:504] T 2be6b563b5e94502afca651b2d803a3c P 76bbe52cb54d4f14b4a8eb90182de87f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } } peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } }
W20260926 03:31:15.872957 1863 tablet.cc:2369] T 2be6b563b5e94502afca651b2d803a3c P 171b9f87cc9d4281838397ec8c90d583: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:31:15.873294 2026 leader_election.cc:288] T 2be6b563b5e94502afca651b2d803a3c P 76bbe52cb54d4f14b4a8eb90182de87f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171b9f87cc9d4281838397ec8c90d583 (127.0.97.2:36863), 4520160883e74babb2b82b6c74c5f265 (127.0.97.1:42837)
W20260926 03:31:15.875002 1724 tablet.cc:2369] T 2be6b563b5e94502afca651b2d803a3c P 4520160883e74babb2b82b6c74c5f265: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:31:15.879812 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:15.873666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44147 (local address 127.0.97.2:36863)
0926 03:31:15.873845 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:15.873853 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:15.873876 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:31:15.873976 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:15.873981 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:15.874067 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:15.874155 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:31:15.874683 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.875637 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.878187 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.878915 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.879066 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:15.879170 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:15.879442 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 03:31:15.879565 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 03:31:15.879841 1890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:15.873484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36863 (local address 127.0.97.3:44147)
0926 03:31:15.873673 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:15.873695 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:15.873817 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:15.874202 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:15.874207 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:15.874231 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:31:15.874491 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:15.874508 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.875794 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.875800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:15.877595 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:15.877605 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.879073 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.879078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:15.879280 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:15.879303 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 03:31:15.879409 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 03:31:15.879551 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 03:31:15.880193 2019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:15.874214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43837 (local address 127.0.97.1:42837)
0926 03:31:15.874455 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:15.874461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:15.874874 (+ 413us) server_negotiation.cc:407] Connection header received
0926 03:31:15.875032 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:15.875038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:15.875145 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:15.875234 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:31:15.875898 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.876953 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.878523 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.879272 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.879426 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:15.879532 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:15.879809 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 03:31:15.879963 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 03:31:15.880626 2029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:15.873853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42837 (local address 127.0.97.3:43837)
0926 03:31:15.874756 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:15.874776 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:15.874895 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:15.875282 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:15.875286 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:15.875312 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:31:15.875698 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:15.875713 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.877082 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.877086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:15.878369 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:15.878380 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.879499 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.879505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:15.879654 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:15.879676 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 03:31:15.879786 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 03:31:15.880288 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":569,"threads_started":1}
I20260926 03:31:15.880918 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2be6b563b5e94502afca651b2d803a3c" candidate_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171b9f87cc9d4281838397ec8c90d583" is_pre_election: true
I20260926 03:31:15.881488 1812 raft_consensus.cc:2454] T 2be6b563b5e94502afca651b2d803a3c P 171b9f87cc9d4281838397ec8c90d583 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76bbe52cb54d4f14b4a8eb90182de87f in term 0.
I20260926 03:31:15.881685 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2be6b563b5e94502afca651b2d803a3c" candidate_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4520160883e74babb2b82b6c74c5f265" is_pre_election: true
I20260926 03:31:15.882191 1885 leader_election.cc:302] T 2be6b563b5e94502afca651b2d803a3c P 76bbe52cb54d4f14b4a8eb90182de87f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171b9f87cc9d4281838397ec8c90d583, 76bbe52cb54d4f14b4a8eb90182de87f; no voters:
I20260926 03:31:15.882269 1673 raft_consensus.cc:2454] T 2be6b563b5e94502afca651b2d803a3c P 4520160883e74babb2b82b6c74c5f265 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76bbe52cb54d4f14b4a8eb90182de87f in term 0.
I20260926 03:31:15.882562 2026 raft_consensus.cc:2781] T 2be6b563b5e94502afca651b2d803a3c P 76bbe52cb54d4f14b4a8eb90182de87f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:31:15.882737 2026 raft_consensus.cc:482] T 2be6b563b5e94502afca651b2d803a3c P 76bbe52cb54d4f14b4a8eb90182de87f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:31:15.882871 2026 raft_consensus.cc:3037] T 2be6b563b5e94502afca651b2d803a3c P 76bbe52cb54d4f14b4a8eb90182de87f [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:15.885164 2026 raft_consensus.cc:504] T 2be6b563b5e94502afca651b2d803a3c P 76bbe52cb54d4f14b4a8eb90182de87f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } } peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } }
I20260926 03:31:15.885732 2026 leader_election.cc:288] T 2be6b563b5e94502afca651b2d803a3c P 76bbe52cb54d4f14b4a8eb90182de87f [CANDIDATE]: Term 1 election: Requested vote from peers 171b9f87cc9d4281838397ec8c90d583 (127.0.97.2:36863), 4520160883e74babb2b82b6c74c5f265 (127.0.97.1:42837)
I20260926 03:31:15.886228 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2be6b563b5e94502afca651b2d803a3c" candidate_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171b9f87cc9d4281838397ec8c90d583"
I20260926 03:31:15.886276 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2be6b563b5e94502afca651b2d803a3c" candidate_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4520160883e74babb2b82b6c74c5f265"
I20260926 03:31:15.886415 1812 raft_consensus.cc:3037] T 2be6b563b5e94502afca651b2d803a3c P 171b9f87cc9d4281838397ec8c90d583 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:15.886484 1673 raft_consensus.cc:3037] T 2be6b563b5e94502afca651b2d803a3c P 4520160883e74babb2b82b6c74c5f265 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:15.888547 1812 raft_consensus.cc:2454] T 2be6b563b5e94502afca651b2d803a3c P 171b9f87cc9d4281838397ec8c90d583 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76bbe52cb54d4f14b4a8eb90182de87f in term 1.
I20260926 03:31:15.888707 1673 raft_consensus.cc:2454] T 2be6b563b5e94502afca651b2d803a3c P 4520160883e74babb2b82b6c74c5f265 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76bbe52cb54d4f14b4a8eb90182de87f in term 1.
I20260926 03:31:15.889029 1885 leader_election.cc:302] T 2be6b563b5e94502afca651b2d803a3c P 76bbe52cb54d4f14b4a8eb90182de87f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171b9f87cc9d4281838397ec8c90d583, 76bbe52cb54d4f14b4a8eb90182de87f; no voters:
I20260926 03:31:15.889314 2026 raft_consensus.cc:2781] T 2be6b563b5e94502afca651b2d803a3c P 76bbe52cb54d4f14b4a8eb90182de87f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:31:15.889734 2026 raft_consensus.cc:686] T 2be6b563b5e94502afca651b2d803a3c P 76bbe52cb54d4f14b4a8eb90182de87f [term 1 LEADER]: Becoming Leader. State: Replica: 76bbe52cb54d4f14b4a8eb90182de87f, State: Running, Role: LEADER
I20260926 03:31:15.890226 2026 consensus_queue.cc:237] T 2be6b563b5e94502afca651b2d803a3c P 76bbe52cb54d4f14b4a8eb90182de87f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } } peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } }
W20260926 03:31:15.891289 2002 tablet.cc:2369] T 2be6b563b5e94502afca651b2d803a3c P 76bbe52cb54d4f14b4a8eb90182de87f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:31:15.895330 1514 catalog_manager.cc:5495] T 2be6b563b5e94502afca651b2d803a3c P 76bbe52cb54d4f14b4a8eb90182de87f reported cstate change: term changed from 0 to 1, leader changed from <none> to 76bbe52cb54d4f14b4a8eb90182de87f (127.0.97.3). New cstate: current_term: 1 leader_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } health_report { overall_health: UNKNOWN } } }
I20260926 03:31:15.930465 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:15.924619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47744 (local address 127.0.97.3:40979)
0926 03:31:15.924842 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:15.924849 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:15.924998 (+ 149us) server_negotiation.cc:407] Connection header received
0926 03:31:15.925372 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:15.925376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:15.925454 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:15.925601 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:31:15.925975 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.926865 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.928193 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:15.928659 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:15.928784 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:15.929293 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:31:15.929803 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:31:15.929974 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:15.930111 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 03:31:15.930191 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:31:15.939679 1673 raft_consensus.cc:1261] T 2be6b563b5e94502afca651b2d803a3c P 4520160883e74babb2b82b6c74c5f265 [term 1 FOLLOWER]: Refusing update from remote peer 76bbe52cb54d4f14b4a8eb90182de87f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:31:15.939679 1812 raft_consensus.cc:1261] T 2be6b563b5e94502afca651b2d803a3c P 171b9f87cc9d4281838397ec8c90d583 [term 1 FOLLOWER]: Refusing update from remote peer 76bbe52cb54d4f14b4a8eb90182de87f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:31:15.940860 2030 consensus_queue.cc:1035] T 2be6b563b5e94502afca651b2d803a3c P 76bbe52cb54d4f14b4a8eb90182de87f [LEADER]: Connected to new peer: Peer: permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:31:15.941331 2026 consensus_queue.cc:1035] T 2be6b563b5e94502afca651b2d803a3c P 76bbe52cb54d4f14b4a8eb90182de87f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:31:15.954566 2033 mvcc.cc:205] Tried to move back new op lower bound from 7333451677435592704 to 7333451677250490368. Current Snapshot: MvccSnapshot[applied={T|T < 7333451677435592704}]
I20260926 03:31:15.974978 1514 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:31:15.975268 1514 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:31:15.979295 1514 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0134055fc0949a1967e0700d4c50ab4: Sending DeleteTablet for 3 replicas of tablet 2be6b563b5e94502afca651b2d803a3c
I20260926 03:31:15.980690 1792 tablet_service.cc:1514] Processing DeleteTablet for tablet 2be6b563b5e94502afca651b2d803a3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:31:15 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60508
I20260926 03:31:15.980700 1931 tablet_service.cc:1514] Processing DeleteTablet for tablet 2be6b563b5e94502afca651b2d803a3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:31:15 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47742
I20260926 03:31:15.981062 1653 tablet_service.cc:1514] Processing DeleteTablet for tablet 2be6b563b5e94502afca651b2d803a3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:31:15 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47492
I20260926 03:31:15.984599 1516 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:31:15.985550 1516 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:31:15.998054 1930 tablet_service.cc:1467] Processing CreateTablet for tablet e77d6c2c6286403f921c780ea8e1ad09 (DEFAULT_TABLE table=test-table [id=9a32fa3f25fd4e129ddb9dadd940a473]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:31:15.998054 1652 tablet_service.cc:1467] Processing CreateTablet for tablet e77d6c2c6286403f921c780ea8e1ad09 (DEFAULT_TABLE table=test-table [id=9a32fa3f25fd4e129ddb9dadd940a473]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:31:15.998641 1652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e77d6c2c6286403f921c780ea8e1ad09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:15.998652 1930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e77d6c2c6286403f921c780ea8e1ad09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:16.001106 1791 tablet_service.cc:1467] Processing CreateTablet for tablet e77d6c2c6286403f921c780ea8e1ad09 (DEFAULT_TABLE table=test-table [id=9a32fa3f25fd4e129ddb9dadd940a473]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:31:16.001722 1791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e77d6c2c6286403f921c780ea8e1ad09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:16.006331 2044 tablet_replica.cc:330] T 2be6b563b5e94502afca651b2d803a3c P 171b9f87cc9d4281838397ec8c90d583: stopping tablet replica
I20260926 03:31:16.006484 2045 tablet_replica.cc:330] T 2be6b563b5e94502afca651b2d803a3c P 4520160883e74babb2b82b6c74c5f265: stopping tablet replica
I20260926 03:31:16.006837 2045 raft_consensus.cc:2229] T 2be6b563b5e94502afca651b2d803a3c P 4520160883e74babb2b82b6c74c5f265 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:31:16.006839 2044 raft_consensus.cc:2229] T 2be6b563b5e94502afca651b2d803a3c P 171b9f87cc9d4281838397ec8c90d583 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:31:16.007330 2044 raft_consensus.cc:2258] T 2be6b563b5e94502afca651b2d803a3c P 171b9f87cc9d4281838397ec8c90d583 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:31:16.007329 2045 raft_consensus.cc:2258] T 2be6b563b5e94502afca651b2d803a3c P 4520160883e74babb2b82b6c74c5f265 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:31:16.010721 2021 tablet_bootstrap.cc:492] T e77d6c2c6286403f921c780ea8e1ad09 P 4520160883e74babb2b82b6c74c5f265: Bootstrap starting.
I20260926 03:31:16.012430 2021 tablet_bootstrap.cc:654] T e77d6c2c6286403f921c780ea8e1ad09 P 4520160883e74babb2b82b6c74c5f265: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:16.013547 2020 tablet_bootstrap.cc:492] T e77d6c2c6286403f921c780ea8e1ad09 P 171b9f87cc9d4281838397ec8c90d583: Bootstrap starting.
I20260926 03:31:16.014139 2022 tablet_bootstrap.cc:492] T e77d6c2c6286403f921c780ea8e1ad09 P 76bbe52cb54d4f14b4a8eb90182de87f: Bootstrap starting.
I20260926 03:31:16.014662 2043 tablet_replica.cc:330] T 2be6b563b5e94502afca651b2d803a3c P 76bbe52cb54d4f14b4a8eb90182de87f: stopping tablet replica
I20260926 03:31:16.015064 2020 tablet_bootstrap.cc:654] T e77d6c2c6286403f921c780ea8e1ad09 P 171b9f87cc9d4281838397ec8c90d583: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:16.015305 2043 raft_consensus.cc:2229] T 2be6b563b5e94502afca651b2d803a3c P 76bbe52cb54d4f14b4a8eb90182de87f [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:31:16.015823 2022 tablet_bootstrap.cc:654] T e77d6c2c6286403f921c780ea8e1ad09 P 76bbe52cb54d4f14b4a8eb90182de87f: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:16.016026 2043 raft_consensus.cc:2258] T 2be6b563b5e94502afca651b2d803a3c P 76bbe52cb54d4f14b4a8eb90182de87f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:31:16.019444 2043 ts_tablet_manager.cc:1905] T 2be6b563b5e94502afca651b2d803a3c P 76bbe52cb54d4f14b4a8eb90182de87f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:31:16.020402 2044 ts_tablet_manager.cc:1905] T 2be6b563b5e94502afca651b2d803a3c P 171b9f87cc9d4281838397ec8c90d583: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:31:16.021934 2045 ts_tablet_manager.cc:1905] T 2be6b563b5e94502afca651b2d803a3c P 4520160883e74babb2b82b6c74c5f265: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:31:16.030830 2043 ts_tablet_manager.cc:1918] T 2be6b563b5e94502afca651b2d803a3c P 76bbe52cb54d4f14b4a8eb90182de87f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:31:16.031626 2020 tablet_bootstrap.cc:492] T e77d6c2c6286403f921c780ea8e1ad09 P 171b9f87cc9d4281838397ec8c90d583: No bootstrap required, opened a new log
I20260926 03:31:16.031864 2020 ts_tablet_manager.cc:1397] T e77d6c2c6286403f921c780ea8e1ad09 P 171b9f87cc9d4281838397ec8c90d583: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.007s
I20260926 03:31:16.033027 2020 raft_consensus.cc:348] T e77d6c2c6286403f921c780ea8e1ad09 P 171b9f87cc9d4281838397ec8c90d583 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } } peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } }
I20260926 03:31:16.033460 2020 raft_consensus.cc:374] T e77d6c2c6286403f921c780ea8e1ad09 P 171b9f87cc9d4281838397ec8c90d583 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:16.033607 2020 raft_consensus.cc:729] T e77d6c2c6286403f921c780ea8e1ad09 P 171b9f87cc9d4281838397ec8c90d583 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171b9f87cc9d4281838397ec8c90d583, State: Initialized, Role: FOLLOWER
I20260926 03:31:16.034021 2020 consensus_queue.cc:260] T e77d6c2c6286403f921c780ea8e1ad09 P 171b9f87cc9d4281838397ec8c90d583 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } } peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } }
I20260926 03:31:16.035722 2020 ts_tablet_manager.cc:1428] T e77d6c2c6286403f921c780ea8e1ad09 P 171b9f87cc9d4281838397ec8c90d583: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260926 03:31:16.036810 2043 log.cc:1196] T 2be6b563b5e94502afca651b2d803a3c P 76bbe52cb54d4f14b4a8eb90182de87f: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-2/wal/wals/2be6b563b5e94502afca651b2d803a3c
I20260926 03:31:16.039245 2045 ts_tablet_manager.cc:1918] T 2be6b563b5e94502afca651b2d803a3c P 4520160883e74babb2b82b6c74c5f265: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:31:16.040275 2043 ts_tablet_manager.cc:1939] T 2be6b563b5e94502afca651b2d803a3c P 76bbe52cb54d4f14b4a8eb90182de87f: Deleting consensus metadata
I20260926 03:31:16.040962 2045 log.cc:1196] T 2be6b563b5e94502afca651b2d803a3c P 4520160883e74babb2b82b6c74c5f265: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-0/wal/wals/2be6b563b5e94502afca651b2d803a3c
I20260926 03:31:16.041756 2045 ts_tablet_manager.cc:1939] T 2be6b563b5e94502afca651b2d803a3c P 4520160883e74babb2b82b6c74c5f265: Deleting consensus metadata
I20260926 03:31:16.042565 2021 tablet_bootstrap.cc:492] T e77d6c2c6286403f921c780ea8e1ad09 P 4520160883e74babb2b82b6c74c5f265: No bootstrap required, opened a new log
I20260926 03:31:16.042727 2021 ts_tablet_manager.cc:1397] T e77d6c2c6286403f921c780ea8e1ad09 P 4520160883e74babb2b82b6c74c5f265: Time spent bootstrapping tablet: real 0.032s user 0.001s sys 0.010s
I20260926 03:31:16.042909 2022 tablet_bootstrap.cc:492] T e77d6c2c6286403f921c780ea8e1ad09 P 76bbe52cb54d4f14b4a8eb90182de87f: No bootstrap required, opened a new log
I20260926 03:31:16.043089 2022 ts_tablet_manager.cc:1397] T e77d6c2c6286403f921c780ea8e1ad09 P 76bbe52cb54d4f14b4a8eb90182de87f: Time spent bootstrapping tablet: real 0.029s user 0.016s sys 0.005s
I20260926 03:31:16.043598 2021 raft_consensus.cc:348] T e77d6c2c6286403f921c780ea8e1ad09 P 4520160883e74babb2b82b6c74c5f265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } } peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } }
I20260926 03:31:16.043735 1503 catalog_manager.cc:4841] TS 4520160883e74babb2b82b6c74c5f265 (127.0.97.1:42837): tablet 2be6b563b5e94502afca651b2d803a3c (table test-table [id=62c3176758c644c08f80c85aa258db94]) successfully deleted
I20260926 03:31:16.043852 2021 raft_consensus.cc:374] T e77d6c2c6286403f921c780ea8e1ad09 P 4520160883e74babb2b82b6c74c5f265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:16.043955 2021 raft_consensus.cc:729] T e77d6c2c6286403f921c780ea8e1ad09 P 4520160883e74babb2b82b6c74c5f265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4520160883e74babb2b82b6c74c5f265, State: Initialized, Role: FOLLOWER
I20260926 03:31:16.043920 2022 raft_consensus.cc:348] T e77d6c2c6286403f921c780ea8e1ad09 P 76bbe52cb54d4f14b4a8eb90182de87f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } } peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } }
I20260926 03:31:16.044077 1502 catalog_manager.cc:4841] TS 76bbe52cb54d4f14b4a8eb90182de87f (127.0.97.3:40979): tablet 2be6b563b5e94502afca651b2d803a3c (table test-table [id=62c3176758c644c08f80c85aa258db94]) successfully deleted
I20260926 03:31:16.044147 2022 raft_consensus.cc:374] T e77d6c2c6286403f921c780ea8e1ad09 P 76bbe52cb54d4f14b4a8eb90182de87f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:16.044240 2022 raft_consensus.cc:729] T e77d6c2c6286403f921c780ea8e1ad09 P 76bbe52cb54d4f14b4a8eb90182de87f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76bbe52cb54d4f14b4a8eb90182de87f, State: Initialized, Role: FOLLOWER
I20260926 03:31:16.044288 2021 consensus_queue.cc:260] T e77d6c2c6286403f921c780ea8e1ad09 P 4520160883e74babb2b82b6c74c5f265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } } peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } }
I20260926 03:31:16.044534 2022 consensus_queue.cc:260] T e77d6c2c6286403f921c780ea8e1ad09 P 76bbe52cb54d4f14b4a8eb90182de87f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } } peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } }
I20260926 03:31:16.045229 2022 ts_tablet_manager.cc:1428] T e77d6c2c6286403f921c780ea8e1ad09 P 76bbe52cb54d4f14b4a8eb90182de87f: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 03:31:16.048064 2044 ts_tablet_manager.cc:1918] T 2be6b563b5e94502afca651b2d803a3c P 171b9f87cc9d4281838397ec8c90d583: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:31:16.048278 2044 log.cc:1196] T 2be6b563b5e94502afca651b2d803a3c P 171b9f87cc9d4281838397ec8c90d583: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-1/wal/wals/2be6b563b5e94502afca651b2d803a3c
I20260926 03:31:16.048923 2044 ts_tablet_manager.cc:1939] T 2be6b563b5e94502afca651b2d803a3c P 171b9f87cc9d4281838397ec8c90d583: Deleting consensus metadata
I20260926 03:31:16.050794 1503 catalog_manager.cc:4841] TS 171b9f87cc9d4281838397ec8c90d583 (127.0.97.2:36863): tablet 2be6b563b5e94502afca651b2d803a3c (table test-table [id=62c3176758c644c08f80c85aa258db94]) successfully deleted
I20260926 03:31:16.051891 2021 ts_tablet_manager.cc:1428] T e77d6c2c6286403f921c780ea8e1ad09 P 4520160883e74babb2b82b6c74c5f265: Time spent starting tablet: real 0.009s user 0.001s sys 0.002s
I20260926 03:31:16.070602 2028 raft_consensus.cc:482] T e77d6c2c6286403f921c780ea8e1ad09 P 4520160883e74babb2b82b6c74c5f265 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:31:16.070853 2028 raft_consensus.cc:504] T e77d6c2c6286403f921c780ea8e1ad09 P 4520160883e74babb2b82b6c74c5f265 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } } peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } }
I20260926 03:31:16.072777 2028 leader_election.cc:288] T e77d6c2c6286403f921c780ea8e1ad09 P 4520160883e74babb2b82b6c74c5f265 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171b9f87cc9d4281838397ec8c90d583 (127.0.97.2:36863), 76bbe52cb54d4f14b4a8eb90182de87f (127.0.97.3:40979)
I20260926 03:31:16.081341 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:16.073408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53409 (local address 127.0.97.2:36863)
0926 03:31:16.073596 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:16.073602 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:16.073868 (+ 266us) server_negotiation.cc:407] Connection header received
0926 03:31:16.074194 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:16.074202 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:16.074306 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:16.074401 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:31:16.075011 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:16.075881 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:16.079085 (+ 3204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:16.080331 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:16.080461 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:16.080598 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:16.080902 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 03:31:16.081068 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 03:31:16.081480 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:16.073026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36863 (local address 127.0.97.1:53409)
0926 03:31:16.073767 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:16.073787 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:16.073978 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:16.074427 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:16.074431 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:16.074461 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:31:16.074810 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:16.074821 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:16.076021 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:16.076028 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:16.078915 (+ 2887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:16.078930 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:16.080555 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:16.080608 (+ 53us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:16.080738 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:16.080765 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 03:31:16.080876 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 03:31:16.081012 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"mutex_wait_us":109,"thread_start_us":413,"threads_started":1}
I20260926 03:31:16.082172 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e77d6c2c6286403f921c780ea8e1ad09" candidate_uuid: "4520160883e74babb2b82b6c74c5f265" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171b9f87cc9d4281838397ec8c90d583" is_pre_election: true
I20260926 03:31:16.082489 1811 raft_consensus.cc:2454] T e77d6c2c6286403f921c780ea8e1ad09 P 171b9f87cc9d4281838397ec8c90d583 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4520160883e74babb2b82b6c74c5f265 in term 0.
I20260926 03:31:16.083164 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:16.073134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37579 (local address 127.0.97.3:40979)
0926 03:31:16.073309 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:16.073317 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:16.074339 (+ 1022us) server_negotiation.cc:407] Connection header received
0926 03:31:16.074545 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:16.074551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:16.074621 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:16.074670 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:31:16.075384 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:16.076544 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:16.080695 (+ 4151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:16.082015 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:16.082145 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:16.082242 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:16.082719 (+ 477us) server_negotiation.cc:299] Negotiation successful
0926 03:31:16.082866 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 03:31:16.083073 1606 leader_election.cc:302] T e77d6c2c6286403f921c780ea8e1ad09 P 4520160883e74babb2b82b6c74c5f265 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171b9f87cc9d4281838397ec8c90d583, 4520160883e74babb2b82b6c74c5f265; no voters:
I20260926 03:31:16.083215 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:16.073031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40979 (local address 127.0.97.1:37579)
0926 03:31:16.074221 (+ 1190us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:16.074240 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:16.074363 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:16.074824 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:16.074827 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:16.074846 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:31:16.075224 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:16.075235 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:16.076720 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:16.076728 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:16.078915 (+ 2187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:16.078930 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:16.082309 (+ 3379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:16.082316 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:16.082496 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:16.082545 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 03:31:16.082684 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 03:31:16.082823 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1045,"mutex_wait_us":125,"thread_start_us":759,"threads_started":1}
I20260926 03:31:16.083527 2028 raft_consensus.cc:2781] T e77d6c2c6286403f921c780ea8e1ad09 P 4520160883e74babb2b82b6c74c5f265 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:31:16.083652 2028 raft_consensus.cc:482] T e77d6c2c6286403f921c780ea8e1ad09 P 4520160883e74babb2b82b6c74c5f265 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:31:16.083738 2028 raft_consensus.cc:3037] T e77d6c2c6286403f921c780ea8e1ad09 P 4520160883e74babb2b82b6c74c5f265 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:16.084797 1951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e77d6c2c6286403f921c780ea8e1ad09" candidate_uuid: "4520160883e74babb2b82b6c74c5f265" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" is_pre_election: true
I20260926 03:31:16.085278 1951 raft_consensus.cc:2454] T e77d6c2c6286403f921c780ea8e1ad09 P 76bbe52cb54d4f14b4a8eb90182de87f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4520160883e74babb2b82b6c74c5f265 in term 0.
I20260926 03:31:16.087496 2028 raft_consensus.cc:504] T e77d6c2c6286403f921c780ea8e1ad09 P 4520160883e74babb2b82b6c74c5f265 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } } peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } }
I20260926 03:31:16.088212 2028 leader_election.cc:288] T e77d6c2c6286403f921c780ea8e1ad09 P 4520160883e74babb2b82b6c74c5f265 [CANDIDATE]: Term 1 election: Requested vote from peers 171b9f87cc9d4281838397ec8c90d583 (127.0.97.2:36863), 76bbe52cb54d4f14b4a8eb90182de87f (127.0.97.3:40979)
I20260926 03:31:16.088775 1951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e77d6c2c6286403f921c780ea8e1ad09" candidate_uuid: "4520160883e74babb2b82b6c74c5f265" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76bbe52cb54d4f14b4a8eb90182de87f"
I20260926 03:31:16.088755 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e77d6c2c6286403f921c780ea8e1ad09" candidate_uuid: "4520160883e74babb2b82b6c74c5f265" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171b9f87cc9d4281838397ec8c90d583"
I20260926 03:31:16.088958 1811 raft_consensus.cc:3037] T e77d6c2c6286403f921c780ea8e1ad09 P 171b9f87cc9d4281838397ec8c90d583 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:16.088980 1951 raft_consensus.cc:3037] T e77d6c2c6286403f921c780ea8e1ad09 P 76bbe52cb54d4f14b4a8eb90182de87f [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:16.091240 1951 raft_consensus.cc:2454] T e77d6c2c6286403f921c780ea8e1ad09 P 76bbe52cb54d4f14b4a8eb90182de87f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4520160883e74babb2b82b6c74c5f265 in term 1.
I20260926 03:31:16.091722 1605 leader_election.cc:302] T e77d6c2c6286403f921c780ea8e1ad09 P 4520160883e74babb2b82b6c74c5f265 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4520160883e74babb2b82b6c74c5f265, 76bbe52cb54d4f14b4a8eb90182de87f; no voters:
I20260926 03:31:16.091907 1811 raft_consensus.cc:2454] T e77d6c2c6286403f921c780ea8e1ad09 P 171b9f87cc9d4281838397ec8c90d583 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4520160883e74babb2b82b6c74c5f265 in term 1.
I20260926 03:31:16.092051 2028 raft_consensus.cc:2781] T e77d6c2c6286403f921c780ea8e1ad09 P 4520160883e74babb2b82b6c74c5f265 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:31:16.092614 2028 raft_consensus.cc:686] T e77d6c2c6286403f921c780ea8e1ad09 P 4520160883e74babb2b82b6c74c5f265 [term 1 LEADER]: Becoming Leader. State: Replica: 4520160883e74babb2b82b6c74c5f265, State: Running, Role: LEADER
I20260926 03:31:16.092998 2028 consensus_queue.cc:237] T e77d6c2c6286403f921c780ea8e1ad09 P 4520160883e74babb2b82b6c74c5f265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } } peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } }
I20260926 03:31:16.096305 1516 catalog_manager.cc:5495] T e77d6c2c6286403f921c780ea8e1ad09 P 4520160883e74babb2b82b6c74c5f265 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4520160883e74babb2b82b6c74c5f265 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "4520160883e74babb2b82b6c74c5f265" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } health_report { overall_health: UNKNOWN } } }
Sep 26 03:31:16 dist-test-slave-dt7p krb5kdc[1471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:31:16.142161 1611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:16.127493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60156 (local address 127.0.97.62:41867)
0926 03:31:16.127701 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:16.127709 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:16.128280 (+ 571us) server_negotiation.cc:407] Connection header received
0926 03:31:16.128489 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:16.128499 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:16.128549 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:16.128702 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:16.130242 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:16.130948 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:16.132211 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:16.132727 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:16.132861 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:16.136132 (+ 3271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:16.136137 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:16.136141 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:16.136169 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:16.139104 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:16.139858 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:16.139862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:16.139864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:16.139976 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:16.140408 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:16.140416 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:16.140422 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:16.140827 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:16.141069 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:16.141534 (+ 465us) server_negotiation.cc:299] Negotiation successful
0926 03:31:16.141745 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 03:31:16.142295 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:16.127493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41867 (local address 127.0.0.1:60156)
0926 03:31:16.128165 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:16.128186 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:16.128361 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:16.128719 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:16.128724 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:16.129414 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:16.130091 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:16.130102 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:16.131076 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:16.131084 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:16.132052 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:16.132064 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:16.132898 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:16.132903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:16.133064 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:16.133776 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:16.133794 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:16.135938 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:16.139247 (+ 3309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:16.139260 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:16.139269 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:16.139677 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:16.140148 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:16.140153 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:16.140156 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:16.140302 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:16.141063 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:16.141071 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:16.141280 (+ 209us) client_negotiation.cc:769] Sending connection context
0926 03:31:16.141450 (+ 170us) client_negotiation.cc:240] Negotiation successful
0926 03:31:16.141881 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":145,"thread_start_us":384,"threads_started":1}
I20260926 03:31:16.148604 1516 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60156:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 03:31:16.149729 1516 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:31:16.156876 1653 tablet_service.cc:1467] Processing CreateTablet for tablet a80befdd5e984c26bce5d409da7a3c73 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=106112582aea40b2a63a5f198b079c93]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:31:16.157234 1791 tablet_service.cc:1467] Processing CreateTablet for tablet a80befdd5e984c26bce5d409da7a3c73 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=106112582aea40b2a63a5f198b079c93]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:31:16.157322 1931 tablet_service.cc:1467] Processing CreateTablet for tablet a80befdd5e984c26bce5d409da7a3c73 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=106112582aea40b2a63a5f198b079c93]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:31:16.157491 1653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a80befdd5e984c26bce5d409da7a3c73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:16.157752 1791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a80befdd5e984c26bce5d409da7a3c73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:16.157793 1931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a80befdd5e984c26bce5d409da7a3c73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:16.165333 2022 tablet_bootstrap.cc:492] T a80befdd5e984c26bce5d409da7a3c73 P 76bbe52cb54d4f14b4a8eb90182de87f: Bootstrap starting.
I20260926 03:31:16.166221 2021 tablet_bootstrap.cc:492] T a80befdd5e984c26bce5d409da7a3c73 P 4520160883e74babb2b82b6c74c5f265: Bootstrap starting.
I20260926 03:31:16.167155 2022 tablet_bootstrap.cc:654] T a80befdd5e984c26bce5d409da7a3c73 P 76bbe52cb54d4f14b4a8eb90182de87f: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:16.168347 2021 tablet_bootstrap.cc:654] T a80befdd5e984c26bce5d409da7a3c73 P 4520160883e74babb2b82b6c74c5f265: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:16.168989 2020 tablet_bootstrap.cc:492] T a80befdd5e984c26bce5d409da7a3c73 P 171b9f87cc9d4281838397ec8c90d583: Bootstrap starting.
I20260926 03:31:16.171164 2020 tablet_bootstrap.cc:654] T a80befdd5e984c26bce5d409da7a3c73 P 171b9f87cc9d4281838397ec8c90d583: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:16.174024 2022 tablet_bootstrap.cc:492] T a80befdd5e984c26bce5d409da7a3c73 P 76bbe52cb54d4f14b4a8eb90182de87f: No bootstrap required, opened a new log
I20260926 03:31:16.174229 2022 ts_tablet_manager.cc:1397] T a80befdd5e984c26bce5d409da7a3c73 P 76bbe52cb54d4f14b4a8eb90182de87f: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20260926 03:31:16.175089 2021 tablet_bootstrap.cc:492] T a80befdd5e984c26bce5d409da7a3c73 P 4520160883e74babb2b82b6c74c5f265: No bootstrap required, opened a new log
I20260926 03:31:16.175040 2022 raft_consensus.cc:348] T a80befdd5e984c26bce5d409da7a3c73 P 76bbe52cb54d4f14b4a8eb90182de87f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } } peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } }
I20260926 03:31:16.175253 2020 tablet_bootstrap.cc:492] T a80befdd5e984c26bce5d409da7a3c73 P 171b9f87cc9d4281838397ec8c90d583: No bootstrap required, opened a new log
I20260926 03:31:16.175304 2022 raft_consensus.cc:374] T a80befdd5e984c26bce5d409da7a3c73 P 76bbe52cb54d4f14b4a8eb90182de87f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:16.175400 2022 raft_consensus.cc:729] T a80befdd5e984c26bce5d409da7a3c73 P 76bbe52cb54d4f14b4a8eb90182de87f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76bbe52cb54d4f14b4a8eb90182de87f, State: Initialized, Role: FOLLOWER
I20260926 03:31:16.175421 2020 ts_tablet_manager.cc:1397] T a80befdd5e984c26bce5d409da7a3c73 P 171b9f87cc9d4281838397ec8c90d583: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260926 03:31:16.175534 2021 ts_tablet_manager.cc:1397] T a80befdd5e984c26bce5d409da7a3c73 P 4520160883e74babb2b82b6c74c5f265: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260926 03:31:16.175711 2022 consensus_queue.cc:260] T a80befdd5e984c26bce5d409da7a3c73 P 76bbe52cb54d4f14b4a8eb90182de87f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } } peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } }
I20260926 03:31:16.176113 2020 raft_consensus.cc:348] T a80befdd5e984c26bce5d409da7a3c73 P 171b9f87cc9d4281838397ec8c90d583 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } } peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } }
I20260926 03:31:16.176348 2020 raft_consensus.cc:374] T a80befdd5e984c26bce5d409da7a3c73 P 171b9f87cc9d4281838397ec8c90d583 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:16.176410 2020 raft_consensus.cc:729] T a80befdd5e984c26bce5d409da7a3c73 P 171b9f87cc9d4281838397ec8c90d583 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171b9f87cc9d4281838397ec8c90d583, State: Initialized, Role: FOLLOWER
I20260926 03:31:16.176594 2026 tablet_replica.cc:439] T a80befdd5e984c26bce5d409da7a3c73 P 76bbe52cb54d4f14b4a8eb90182de87f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } } peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } } }
I20260926 03:31:16.176643 2020 consensus_queue.cc:260] T a80befdd5e984c26bce5d409da7a3c73 P 171b9f87cc9d4281838397ec8c90d583 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } } peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } }
I20260926 03:31:16.176879 2026 tablet_replica.cc:442] T a80befdd5e984c26bce5d409da7a3c73 P 76bbe52cb54d4f14b4a8eb90182de87f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:31:16.177273 2020 ts_tablet_manager.cc:1428] T a80befdd5e984c26bce5d409da7a3c73 P 171b9f87cc9d4281838397ec8c90d583: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 03:31:16.177079 2027 tablet_replica.cc:439] T a80befdd5e984c26bce5d409da7a3c73 P 171b9f87cc9d4281838397ec8c90d583: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } } peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } } }
I20260926 03:31:16.177175 2021 raft_consensus.cc:348] T a80befdd5e984c26bce5d409da7a3c73 P 4520160883e74babb2b82b6c74c5f265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } } peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } }
I20260926 03:31:16.177451 2027 tablet_replica.cc:442] T a80befdd5e984c26bce5d409da7a3c73 P 171b9f87cc9d4281838397ec8c90d583: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:31:16.177578 2021 raft_consensus.cc:374] T a80befdd5e984c26bce5d409da7a3c73 P 4520160883e74babb2b82b6c74c5f265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:16.177716 2021 raft_consensus.cc:729] T a80befdd5e984c26bce5d409da7a3c73 P 4520160883e74babb2b82b6c74c5f265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4520160883e74babb2b82b6c74c5f265, State: Initialized, Role: FOLLOWER
I20260926 03:31:16.178216 2021 consensus_queue.cc:260] T a80befdd5e984c26bce5d409da7a3c73 P 4520160883e74babb2b82b6c74c5f265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } } peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } }
I20260926 03:31:16.178689 2028 tablet_replica.cc:439] T a80befdd5e984c26bce5d409da7a3c73 P 4520160883e74babb2b82b6c74c5f265: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } } peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } } }
I20260926 03:31:16.178953 2028 tablet_replica.cc:442] T a80befdd5e984c26bce5d409da7a3c73 P 4520160883e74babb2b82b6c74c5f265: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:31:16.179486 2021 ts_tablet_manager.cc:1428] T a80befdd5e984c26bce5d409da7a3c73 P 4520160883e74babb2b82b6c74c5f265: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260926 03:31:16.179908 2022 ts_tablet_manager.cc:1428] T a80befdd5e984c26bce5d409da7a3c73 P 76bbe52cb54d4f14b4a8eb90182de87f: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20260926 03:31:16.191937 2026 raft_consensus.cc:482] T a80befdd5e984c26bce5d409da7a3c73 P 76bbe52cb54d4f14b4a8eb90182de87f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:31:16.192133 2026 raft_consensus.cc:504] T a80befdd5e984c26bce5d409da7a3c73 P 76bbe52cb54d4f14b4a8eb90182de87f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } } peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } }
I20260926 03:31:16.192852 2026 leader_election.cc:288] T a80befdd5e984c26bce5d409da7a3c73 P 76bbe52cb54d4f14b4a8eb90182de87f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4520160883e74babb2b82b6c74c5f265 (127.0.97.1:42837), 171b9f87cc9d4281838397ec8c90d583 (127.0.97.2:36863)
I20260926 03:31:16.193302 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80befdd5e984c26bce5d409da7a3c73" candidate_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4520160883e74babb2b82b6c74c5f265" is_pre_election: true
I20260926 03:31:16.193420 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80befdd5e984c26bce5d409da7a3c73" candidate_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171b9f87cc9d4281838397ec8c90d583" is_pre_election: true
I20260926 03:31:16.193547 1673 raft_consensus.cc:2454] T a80befdd5e984c26bce5d409da7a3c73 P 4520160883e74babb2b82b6c74c5f265 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76bbe52cb54d4f14b4a8eb90182de87f in term 0.
I20260926 03:31:16.193714 1811 raft_consensus.cc:2454] T a80befdd5e984c26bce5d409da7a3c73 P 171b9f87cc9d4281838397ec8c90d583 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76bbe52cb54d4f14b4a8eb90182de87f in term 0.
I20260926 03:31:16.193959 1885 leader_election.cc:302] T a80befdd5e984c26bce5d409da7a3c73 P 76bbe52cb54d4f14b4a8eb90182de87f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4520160883e74babb2b82b6c74c5f265, 76bbe52cb54d4f14b4a8eb90182de87f; no voters:
I20260926 03:31:16.194227 2026 raft_consensus.cc:2781] T a80befdd5e984c26bce5d409da7a3c73 P 76bbe52cb54d4f14b4a8eb90182de87f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:31:16.194365 2026 raft_consensus.cc:482] T a80befdd5e984c26bce5d409da7a3c73 P 76bbe52cb54d4f14b4a8eb90182de87f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:31:16.194499 2026 raft_consensus.cc:3037] T a80befdd5e984c26bce5d409da7a3c73 P 76bbe52cb54d4f14b4a8eb90182de87f [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:16.197059 2026 raft_consensus.cc:504] T a80befdd5e984c26bce5d409da7a3c73 P 76bbe52cb54d4f14b4a8eb90182de87f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } } peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } }
I20260926 03:31:16.197690 2026 leader_election.cc:288] T a80befdd5e984c26bce5d409da7a3c73 P 76bbe52cb54d4f14b4a8eb90182de87f [CANDIDATE]: Term 1 election: Requested vote from peers 4520160883e74babb2b82b6c74c5f265 (127.0.97.1:42837), 171b9f87cc9d4281838397ec8c90d583 (127.0.97.2:36863)
I20260926 03:31:16.198139 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80befdd5e984c26bce5d409da7a3c73" candidate_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4520160883e74babb2b82b6c74c5f265"
I20260926 03:31:16.198213 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80befdd5e984c26bce5d409da7a3c73" candidate_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171b9f87cc9d4281838397ec8c90d583"
I20260926 03:31:16.198348 1673 raft_consensus.cc:3037] T a80befdd5e984c26bce5d409da7a3c73 P 4520160883e74babb2b82b6c74c5f265 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:16.198432 1811 raft_consensus.cc:3037] T a80befdd5e984c26bce5d409da7a3c73 P 171b9f87cc9d4281838397ec8c90d583 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:16.201319 1811 raft_consensus.cc:2454] T a80befdd5e984c26bce5d409da7a3c73 P 171b9f87cc9d4281838397ec8c90d583 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76bbe52cb54d4f14b4a8eb90182de87f in term 1.
I20260926 03:31:16.201340 1673 raft_consensus.cc:2454] T a80befdd5e984c26bce5d409da7a3c73 P 4520160883e74babb2b82b6c74c5f265 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76bbe52cb54d4f14b4a8eb90182de87f in term 1.
I20260926 03:31:16.201699 1885 leader_election.cc:302] T a80befdd5e984c26bce5d409da7a3c73 P 76bbe52cb54d4f14b4a8eb90182de87f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4520160883e74babb2b82b6c74c5f265, 76bbe52cb54d4f14b4a8eb90182de87f; no voters:
I20260926 03:31:16.202059 2026 raft_consensus.cc:2781] T a80befdd5e984c26bce5d409da7a3c73 P 76bbe52cb54d4f14b4a8eb90182de87f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:31:16.202267 2026 raft_consensus.cc:686] T a80befdd5e984c26bce5d409da7a3c73 P 76bbe52cb54d4f14b4a8eb90182de87f [term 1 LEADER]: Becoming Leader. State: Replica: 76bbe52cb54d4f14b4a8eb90182de87f, State: Running, Role: LEADER
I20260926 03:31:16.202569 2026 consensus_queue.cc:237] T a80befdd5e984c26bce5d409da7a3c73 P 76bbe52cb54d4f14b4a8eb90182de87f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } } peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } }
I20260926 03:31:16.204389 2040 tablet_replica.cc:439] T a80befdd5e984c26bce5d409da7a3c73 P 76bbe52cb54d4f14b4a8eb90182de87f: TxnStatusTablet state changed. Reason: New leader 76bbe52cb54d4f14b4a8eb90182de87f. Latest consensus state: current_term: 1 leader_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } } peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } } }
I20260926 03:31:16.204730 2040 tablet_replica.cc:442] T a80befdd5e984c26bce5d409da7a3c73 P 76bbe52cb54d4f14b4a8eb90182de87f: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:31:16.206037 1514 catalog_manager.cc:5495] T a80befdd5e984c26bce5d409da7a3c73 P 76bbe52cb54d4f14b4a8eb90182de87f reported cstate change: term changed from 0 to 1, leader changed from <none> to 76bbe52cb54d4f14b4a8eb90182de87f (127.0.97.3). New cstate: current_term: 1 leader_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } health_report { overall_health: HEALTHY } } }
I20260926 03:31:16.207892 2063 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 03:31:16.208137 2063 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 03:31:16.227133 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:16.221584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47752 (local address 127.0.97.3:40979)
0926 03:31:16.221759 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:16.221765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:16.221853 (+ 88us) server_negotiation.cc:407] Connection header received
0926 03:31:16.222106 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:16.222110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:16.222197 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:16.222273 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:31:16.222663 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:16.223391 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:16.225418 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:16.225845 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:16.225938 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:16.226459 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:31:16.226563 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:31:16.226624 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:16.226821 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 03:31:16.226881 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 03:31:16.227264 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:16.221497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40979 (local address 127.0.0.1:47752)
0926 03:31:16.221757 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:16.221778 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:16.221936 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:16.222323 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:16.222328 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:16.222341 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:31:16.222534 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:16.222545 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:16.223610 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:16.223616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:16.225277 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:16.225292 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:16.225991 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:16.225997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:16.226201 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:16.226286 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:31:16.226657 (+ 371us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:31:16.226675 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 03:31:16.226778 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 03:31:16.226931 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 03:31:16.231357 1811 raft_consensus.cc:1261] T a80befdd5e984c26bce5d409da7a3c73 P 171b9f87cc9d4281838397ec8c90d583 [term 1 FOLLOWER]: Refusing update from remote peer 76bbe52cb54d4f14b4a8eb90182de87f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:31:16.231379 1673 raft_consensus.cc:1261] T a80befdd5e984c26bce5d409da7a3c73 P 4520160883e74babb2b82b6c74c5f265 [term 1 FOLLOWER]: Refusing update from remote peer 76bbe52cb54d4f14b4a8eb90182de87f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:31:16.232295 2040 consensus_queue.cc:1035] T a80befdd5e984c26bce5d409da7a3c73 P 76bbe52cb54d4f14b4a8eb90182de87f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 03:31:16.233008 2026 consensus_queue.cc:1035] T a80befdd5e984c26bce5d409da7a3c73 P 76bbe52cb54d4f14b4a8eb90182de87f [LEADER]: Connected to new peer: Peer: permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:31:16.238312 2027 tablet_replica.cc:439] T a80befdd5e984c26bce5d409da7a3c73 P 171b9f87cc9d4281838397ec8c90d583: TxnStatusTablet state changed. Reason: New leader 76bbe52cb54d4f14b4a8eb90182de87f. Latest consensus state: current_term: 1 leader_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } } peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } } }
I20260926 03:31:16.238631 2027 tablet_replica.cc:442] T a80befdd5e984c26bce5d409da7a3c73 P 171b9f87cc9d4281838397ec8c90d583: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:31:16.242039 2026 tablet_replica.cc:439] T a80befdd5e984c26bce5d409da7a3c73 P 76bbe52cb54d4f14b4a8eb90182de87f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } } peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } } }
I20260926 03:31:16.242370 2026 tablet_replica.cc:442] T a80befdd5e984c26bce5d409da7a3c73 P 76bbe52cb54d4f14b4a8eb90182de87f: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:31:16.243461 2034 mvcc.cc:205] Tried to move back new op lower bound from 7333451678636269568 to 7333451678528421888. Current Snapshot: MvccSnapshot[applied={T|T < 7333451678636269568}]
I20260926 03:31:16.244544 2028 tablet_replica.cc:439] T a80befdd5e984c26bce5d409da7a3c73 P 4520160883e74babb2b82b6c74c5f265: TxnStatusTablet state changed. Reason: New leader 76bbe52cb54d4f14b4a8eb90182de87f. Latest consensus state: current_term: 1 leader_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } } peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } } }
I20260926 03:31:16.245041 2028 tablet_replica.cc:442] T a80befdd5e984c26bce5d409da7a3c73 P 4520160883e74babb2b82b6c74c5f265: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:31:16.245141 2026 tablet_replica.cc:439] T a80befdd5e984c26bce5d409da7a3c73 P 76bbe52cb54d4f14b4a8eb90182de87f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } } peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } } }
I20260926 03:31:16.245364 2026 tablet_replica.cc:442] T a80befdd5e984c26bce5d409da7a3c73 P 76bbe52cb54d4f14b4a8eb90182de87f: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:31:16.247454 2028 tablet_replica.cc:439] T a80befdd5e984c26bce5d409da7a3c73 P 4520160883e74babb2b82b6c74c5f265: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } } peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } } }
I20260926 03:31:16.247740 2028 tablet_replica.cc:442] T a80befdd5e984c26bce5d409da7a3c73 P 4520160883e74babb2b82b6c74c5f265: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:31:16.248101 2027 tablet_replica.cc:439] T a80befdd5e984c26bce5d409da7a3c73 P 171b9f87cc9d4281838397ec8c90d583: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4520160883e74babb2b82b6c74c5f265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42837 } } peers { permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 } } peers { permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 } } }
I20260926 03:31:16.248360 2027 tablet_replica.cc:442] T a80befdd5e984c26bce5d409da7a3c73 P 171b9f87cc9d4281838397ec8c90d583: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:31:16.254863 2019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:16.248391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47496 (local address 127.0.97.1:42837)
0926 03:31:16.248545 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:16.248550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:16.248590 (+ 40us) server_negotiation.cc:407] Connection header received
0926 03:31:16.248688 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:16.248693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:16.248812 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:16.248992 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:31:16.249419 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:16.250577 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:16.252207 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:16.252907 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:16.253037 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:16.253472 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:31:16.254240 (+ 768us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:31:16.254384 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:16.254469 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 03:31:16.254554 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 03:31:16.272886 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:16.266362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57775 (local address 127.0.97.3:40979)
0926 03:31:16.266601 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:16.266609 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:16.267053 (+ 444us) server_negotiation.cc:407] Connection header received
0926 03:31:16.267303 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:16.267314 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:16.267381 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:16.267482 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:31:16.267987 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:16.268940 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:16.270867 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:16.271342 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:16.271482 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:16.271937 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:31:16.272064 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:31:16.272230 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:16.272385 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 03:31:16.272478 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 03:31:16.272964 2069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:16.266204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40979 (local address 127.0.97.1:57775)
0926 03:31:16.266942 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:16.266962 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:16.267171 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:16.267504 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:16.267509 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:16.267520 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:31:16.267821 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:16.267829 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:16.269084 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:16.269091 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:16.270705 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:16.270719 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:16.271538 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:16.271547 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:16.271673 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:16.271735 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:31:16.272252 (+ 517us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:31:16.272270 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 03:31:16.272363 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 03:31:16.272523 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":401,"threads_started":1}
I20260926 03:31:16.283867 1951 raft_consensus.cc:1261] T e77d6c2c6286403f921c780ea8e1ad09 P 76bbe52cb54d4f14b4a8eb90182de87f [term 1 FOLLOWER]: Refusing update from remote peer 4520160883e74babb2b82b6c74c5f265: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:31:16.284500 1811 raft_consensus.cc:1261] T e77d6c2c6286403f921c780ea8e1ad09 P 171b9f87cc9d4281838397ec8c90d583 [term 1 FOLLOWER]: Refusing update from remote peer 4520160883e74babb2b82b6c74c5f265: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:31:16.284713 2051 consensus_queue.cc:1035] T e77d6c2c6286403f921c780ea8e1ad09 P 4520160883e74babb2b82b6c74c5f265 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76bbe52cb54d4f14b4a8eb90182de87f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:31:16.285120 2028 consensus_queue.cc:1035] T e77d6c2c6286403f921c780ea8e1ad09 P 4520160883e74babb2b82b6c74c5f265 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171b9f87cc9d4281838397ec8c90d583" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:31:16.321669 2019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:16.314889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49183 (local address 127.0.97.1:42837)
0926 03:31:16.315048 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:16.315055 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:16.315795 (+ 740us) server_negotiation.cc:407] Connection header received
0926 03:31:16.316032 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:16.316037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:16.316111 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:16.316210 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:31:16.316996 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:16.317935 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:16.319881 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:16.320299 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:16.320439 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:16.320921 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:31:16.321041 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:31:16.321137 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:16.321276 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 03:31:16.321353 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 03:31:16.321805 2073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:16.314835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42837 (local address 127.0.97.3:49183)
0926 03:31:16.315707 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:16.315729 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:16.315874 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:16.316225 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:16.316229 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:16.316244 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:31:16.316844 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:16.316855 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:16.318061 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:16.318066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:16.319761 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:16.319776 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:16.320463 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:16.320470 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:16.320681 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:16.320749 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:31:16.321136 (+ 387us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:31:16.321152 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 03:31:16.321248 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 03:31:16.321387 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":488,"threads_started":1}
I20260926 03:31:16.362186 1516 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:31:16.362435 1516 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:31:16.365900 1516 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0134055fc0949a1967e0700d4c50ab4: Sending DeleteTablet for 3 replicas of tablet e77d6c2c6286403f921c780ea8e1ad09
I20260926 03:31:16.366809 1653 tablet_service.cc:1514] Processing DeleteTablet for tablet e77d6c2c6286403f921c780ea8e1ad09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:31:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47492
I20260926 03:31:16.366962 1931 tablet_service.cc:1514] Processing DeleteTablet for tablet e77d6c2c6286403f921c780ea8e1ad09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:31:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47742
I20260926 03:31:16.367126 2045 tablet_replica.cc:330] T e77d6c2c6286403f921c780ea8e1ad09 P 4520160883e74babb2b82b6c74c5f265: stopping tablet replica
I20260926 03:31:16.367218 2043 tablet_replica.cc:330] T e77d6c2c6286403f921c780ea8e1ad09 P 76bbe52cb54d4f14b4a8eb90182de87f: stopping tablet replica
I20260926 03:31:16.367362 2045 raft_consensus.cc:2229] T e77d6c2c6286403f921c780ea8e1ad09 P 4520160883e74babb2b82b6c74c5f265 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:31:16.367409 2043 raft_consensus.cc:2229] T e77d6c2c6286403f921c780ea8e1ad09 P 76bbe52cb54d4f14b4a8eb90182de87f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:31:16.367620 2043 raft_consensus.cc:2258] T e77d6c2c6286403f921c780ea8e1ad09 P 76bbe52cb54d4f14b4a8eb90182de87f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:31:16.367892 2045 raft_consensus.cc:2258] T e77d6c2c6286403f921c780ea8e1ad09 P 4520160883e74babb2b82b6c74c5f265 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:31:16.368335 1791 tablet_service.cc:1514] Processing DeleteTablet for tablet e77d6c2c6286403f921c780ea8e1ad09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:31:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60508
I20260926 03:31:16.368677 2044 tablet_replica.cc:330] T e77d6c2c6286403f921c780ea8e1ad09 P 171b9f87cc9d4281838397ec8c90d583: stopping tablet replica
I20260926 03:31:16.368867 2044 raft_consensus.cc:2229] T e77d6c2c6286403f921c780ea8e1ad09 P 171b9f87cc9d4281838397ec8c90d583 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:31:16.369060 2044 raft_consensus.cc:2258] T e77d6c2c6286403f921c780ea8e1ad09 P 171b9f87cc9d4281838397ec8c90d583 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:31:16.369154 2043 ts_tablet_manager.cc:1905] T e77d6c2c6286403f921c780ea8e1ad09 P 76bbe52cb54d4f14b4a8eb90182de87f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:31:16.369163 2045 ts_tablet_manager.cc:1905] T e77d6c2c6286403f921c780ea8e1ad09 P 4520160883e74babb2b82b6c74c5f265: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:31:16.378254 2044 ts_tablet_manager.cc:1905] T e77d6c2c6286403f921c780ea8e1ad09 P 171b9f87cc9d4281838397ec8c90d583: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:31:16.379441 2043 ts_tablet_manager.cc:1918] T e77d6c2c6286403f921c780ea8e1ad09 P 76bbe52cb54d4f14b4a8eb90182de87f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:31:16.379770 2043 log.cc:1196] T e77d6c2c6286403f921c780ea8e1ad09 P 76bbe52cb54d4f14b4a8eb90182de87f: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-2/wal/wals/e77d6c2c6286403f921c780ea8e1ad09
I20260926 03:31:16.380446 2043 ts_tablet_manager.cc:1939] T e77d6c2c6286403f921c780ea8e1ad09 P 76bbe52cb54d4f14b4a8eb90182de87f: Deleting consensus metadata
I20260926 03:31:16.381388 2045 ts_tablet_manager.cc:1918] T e77d6c2c6286403f921c780ea8e1ad09 P 4520160883e74babb2b82b6c74c5f265: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:31:16.381556 2045 log.cc:1196] T e77d6c2c6286403f921c780ea8e1ad09 P 4520160883e74babb2b82b6c74c5f265: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-0/wal/wals/e77d6c2c6286403f921c780ea8e1ad09
I20260926 03:31:16.381949 2045 ts_tablet_manager.cc:1939] T e77d6c2c6286403f921c780ea8e1ad09 P 4520160883e74babb2b82b6c74c5f265: Deleting consensus metadata
I20260926 03:31:16.383399 1503 catalog_manager.cc:4841] TS 4520160883e74babb2b82b6c74c5f265 (127.0.97.1:42837): tablet e77d6c2c6286403f921c780ea8e1ad09 (table test-table [id=9a32fa3f25fd4e129ddb9dadd940a473]) successfully deleted
I20260926 03:31:16.385324 1502 catalog_manager.cc:4841] TS 76bbe52cb54d4f14b4a8eb90182de87f (127.0.97.3:40979): tablet e77d6c2c6286403f921c780ea8e1ad09 (table test-table [id=9a32fa3f25fd4e129ddb9dadd940a473]) successfully deleted
I20260926 03:31:16.386127 2044 ts_tablet_manager.cc:1918] T e77d6c2c6286403f921c780ea8e1ad09 P 171b9f87cc9d4281838397ec8c90d583: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:31:16.386396 2044 log.cc:1196] T e77d6c2c6286403f921c780ea8e1ad09 P 171b9f87cc9d4281838397ec8c90d583: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790393463219689-388-0/minicluster-data/ts-1/wal/wals/e77d6c2c6286403f921c780ea8e1ad09
Sep 26 03:31:16 dist-test-slave-dt7p krb5kdc[1471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393475, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:31:16.397120 2044 ts_tablet_manager.cc:1939] T e77d6c2c6286403f921c780ea8e1ad09 P 171b9f87cc9d4281838397ec8c90d583: Deleting consensus metadata
I20260926 03:31:16.398675 1503 catalog_manager.cc:4841] TS 171b9f87cc9d4281838397ec8c90d583 (127.0.97.2:36863): tablet e77d6c2c6286403f921c780ea8e1ad09 (table test-table [id=9a32fa3f25fd4e129ddb9dadd940a473]) successfully deleted
I20260926 03:31:16.402276 2019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:16.383160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47512 (local address 127.0.97.1:42837)
0926 03:31:16.383865 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:16.383875 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:16.384529 (+ 654us) server_negotiation.cc:407] Connection header received
0926 03:31:16.384811 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:16.384816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:16.384898 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:16.385007 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:16.387529 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:16.388282 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:16.389677 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:16.390123 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:16.390279 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:16.394556 (+ 4277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:16.394570 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:16.394586 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:16.394622 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:16.398896 (+ 4274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:16.399895 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:16.399906 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:16.399910 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:16.400076 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:16.400487 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:16.400492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:16.400495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:16.400973 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:16.401326 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:16.401615 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 03:31:16.401887 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488}
W20260926 03:31:16.403046 1718 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47512
I20260926 03:31:16.424042 1611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:16.406780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60172 (local address 127.0.97.62:41867)
0926 03:31:16.406989 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:16.406995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:16.407478 (+ 483us) server_negotiation.cc:407] Connection header received
0926 03:31:16.407699 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:16.407705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:16.407780 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:16.407882 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:16.411040 (+ 3158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:16.412169 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:16.413919 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:16.414401 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:16.414566 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:16.417490 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:16.417497 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:16.417500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:16.417549 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:16.421055 (+ 3506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:16.421910 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:16.421917 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:16.421920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:16.422071 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:16.422476 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:16.422482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:16.422485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:16.422813 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:16.423099 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:16.423433 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 03:31:16.423674 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 03:31:16.424705 1516 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60172
I20260926 03:31:16.426955 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 1587
I20260926 03:31:16.450953 2019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:16.428338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47522 (local address 127.0.97.1:42837)
0926 03:31:16.428514 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:16.428521 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:16.437200 (+ 8679us) server_negotiation.cc:407] Connection header received
0926 03:31:16.437419 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:16.437424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:16.437505 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:16.437618 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:16.438836 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:16.439726 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:16.441300 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:16.441704 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:16.441850 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:16.444191 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:16.444199 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:16.444203 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:16.444244 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:16.447961 (+ 3717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:16.448742 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:16.448748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:16.448752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:16.448902 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:16.449320 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:16.449324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:16.449327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:16.449676 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:16.449915 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:16.450312 (+ 397us) server_negotiation.cc:299] Negotiation successful
0926 03:31:16.450561 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 03:31:16.451542 1718 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47522
W20260926 03:31:16.452167 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 1587: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:31:16.452374 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 1587
I20260926 03:31:16.478214 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 1727
Sep 26 03:31:16 dist-test-slave-dt7p krb5kdc[1471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393475, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:31:16.495420 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:16.479316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60520 (local address 127.0.97.2:36863)
0926 03:31:16.479504 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:16.479510 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:16.479533 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:31:16.479645 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:16.479650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:16.479736 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:16.479850 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:16.481005 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:16.482139 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:16.483572 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:16.484224 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:16.484371 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:16.488602 (+ 4231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:16.488623 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:16.488644 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:16.488719 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:16.492619 (+ 3900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:16.493321 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:16.493331 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:16.493335 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:16.493499 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:16.493839 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:16.493844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:16.493847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:16.494301 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:16.494625 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:16.494774 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 03:31:16.495000 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 03:31:16.496268 1857 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60520
W20260926 03:31:16.497196 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 1727: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:31:16.497419 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 1727
I20260926 03:31:16.516870 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 1866
Sep 26 03:31:16 dist-test-slave-dt7p krb5kdc[1471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393475, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:31:16.532735 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:16.517676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47754 (local address 127.0.97.3:40979)
0926 03:31:16.517831 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:16.517836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:16.518017 (+ 181us) server_negotiation.cc:407] Connection header received
0926 03:31:16.518221 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:16.518226 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:16.518300 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:16.518403 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:16.519626 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:16.520403 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:16.521648 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:16.522097 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:16.522232 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:16.526499 (+ 4267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:16.526511 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:16.526527 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:16.526564 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:16.529888 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:16.530556 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:16.530568 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:16.530571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:16.530705 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:16.531107 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:16.531114 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:16.531117 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:16.531496 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:16.531865 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:16.532086 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 03:31:16.532332 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 03:31:16.533488 1996 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47754
W20260926 03:31:16.534164 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 1866: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:31:16.534354 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 1866
I20260926 03:31:16.556062 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 1484
I20260926 03:31:16.557202 1556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:31:16.672626 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 1484
2026-09-26T03:31:16Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4429 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:31:16 dist-test-slave-dt7p krb5kdc[2085](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:31:16 dist-test-slave-dt7p krb5kdc[2085](info): set up 2 sockets
Sep 26 03:31:16 dist-test-slave-dt7p krb5kdc[2085](info): commencing operation
krb5kdc: starting...
W20260926 03:31:17.764708 388 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:31:17 dist-test-slave-dt7p krb5kdc[2085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:31:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:31:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:31:17.948139 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33565
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:18.270383 2098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:18.270730 2098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:18.270898 2098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:18.278009 2098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:31:18.278153 2098 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 03:31:18.278254 2098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:18.278376 2098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:18.278460 2098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:31:18.278548 2098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:31:18.288178 2098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33565
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:18.289510 2098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:18.290856 2098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:18.302402 2103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:18.302350 2104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:18.302314 2106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:18.303347 2098 server_base.cc:1034] running on GCE node
I20260926 03:31:18.304011 2098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:18.305387 2098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:18.306630 2098 hybrid_clock.cc:648] HybridClock initialized: now 1790393478306578 us; error 32 us; skew 500 ppm
I20260926 03:31:18.306952 2098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:18 dist-test-slave-dt7p krb5kdc[2085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393478, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:18.318046 2098 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:31:18.319504 2098 webserver.cc:460] Webserver started at http://127.0.97.62:41141/ using document root <none> and password file <none>
I20260926 03:31:18.320210 2098 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:18.320336 2098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:18.320742 2098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:18.323640 2098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "726f37f70bde44e699103131c029838d"
format_stamp: "Formatted at 2026-09-26 03:31:18 on dist-test-slave-dt7p"
server_key: "12e81fe6b32095a9166cfb4765493aab"
server_key_iv: "ed5ea89c2a2ba3e4ee7a325157adfd57"
server_key_version: "encryptionkey@0"
I20260926 03:31:18.324396 2098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "726f37f70bde44e699103131c029838d"
format_stamp: "Formatted at 2026-09-26 03:31:18 on dist-test-slave-dt7p"
server_key: "12e81fe6b32095a9166cfb4765493aab"
server_key_iv: "ed5ea89c2a2ba3e4ee7a325157adfd57"
server_key_version: "encryptionkey@0"
I20260926 03:31:18.329506 2098 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 03:31:18.332775 2113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:18.334122 2098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:31:18.334247 2098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "726f37f70bde44e699103131c029838d"
format_stamp: "Formatted at 2026-09-26 03:31:18 on dist-test-slave-dt7p"
server_key: "12e81fe6b32095a9166cfb4765493aab"
server_key_iv: "ed5ea89c2a2ba3e4ee7a325157adfd57"
server_key_version: "encryptionkey@0"
I20260926 03:31:18.334409 2098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:18.347565 2098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:18.351562 2098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:18.351770 2098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:18.373344 2098 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33565
I20260926 03:31:18.373368 2174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33565 every 8 connection(s)
I20260926 03:31:18.374984 2098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:31:18.378679 2175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:18.384452 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 2098
I20260926 03:31:18.384709 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:31:18.385366 388 external_mini_cluster.cc:1428] Setting key 38c235cc990abf833c46d16d4f631081
I20260926 03:31:18.389513 2175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 726f37f70bde44e699103131c029838d: Bootstrap starting.
I20260926 03:31:18.393298 2175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 726f37f70bde44e699103131c029838d: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:18.394528 2175 log.cc:824] T 00000000000000000000000000000000 P 726f37f70bde44e699103131c029838d: Log is configured to *not* fsync() on all Append() calls
I20260926 03:31:18.397680 2175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 726f37f70bde44e699103131c029838d: No bootstrap required, opened a new log
Sep 26 03:31:18 dist-test-slave-dt7p krb5kdc[2085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:31:18.407173 2175 raft_consensus.cc:348] T 00000000000000000000000000000000 P 726f37f70bde44e699103131c029838d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "726f37f70bde44e699103131c029838d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33565 } }
I20260926 03:31:18.407644 2175 raft_consensus.cc:374] T 00000000000000000000000000000000 P 726f37f70bde44e699103131c029838d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:18.407783 2175 raft_consensus.cc:729] T 00000000000000000000000000000000 P 726f37f70bde44e699103131c029838d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 726f37f70bde44e699103131c029838d, State: Initialized, Role: FOLLOWER
I20260926 03:31:18.408453 2175 consensus_queue.cc:260] T 00000000000000000000000000000000 P 726f37f70bde44e699103131c029838d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "726f37f70bde44e699103131c029838d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33565 } }
I20260926 03:31:18.408749 2175 raft_consensus.cc:388] T 00000000000000000000000000000000 P 726f37f70bde44e699103131c029838d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:31:18.408892 2175 raft_consensus.cc:482] T 00000000000000000000000000000000 P 726f37f70bde44e699103131c029838d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:31:18.409090 2175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 726f37f70bde44e699103131c029838d [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:18.409864 2178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:18.387385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53028 (local address 127.0.97.62:33565)
0926 03:31:18.388174 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:18.388188 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:18.388238 (+ 50us) server_negotiation.cc:407] Connection header received
0926 03:31:18.390576 (+ 2338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:18.390604 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:18.391080 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:18.391536 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:18.392814 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:18.394793 (+ 1979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:18.396358 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:18.397376 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:18.397564 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:18.401456 (+ 3892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:18.401467 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:18.401491 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:18.401538 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:18.405460 (+ 3922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:18.406094 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:18.406104 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:18.406115 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:18.406315 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:18.406637 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:18.406641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:18.406644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:18.407315 (+ 671us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:18.407595 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:18.407996 (+ 401us) server_negotiation.cc:299] Negotiation successful
0926 03:31:18.408407 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":326,"threads_started":1}
I20260926 03:31:18.412039 2175 raft_consensus.cc:504] T 00000000000000000000000000000000 P 726f37f70bde44e699103131c029838d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "726f37f70bde44e699103131c029838d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33565 } }
I20260926 03:31:18.412637 2175 leader_election.cc:302] T 00000000000000000000000000000000 P 726f37f70bde44e699103131c029838d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 726f37f70bde44e699103131c029838d; no voters:
I20260926 03:31:18.413197 2175 leader_election.cc:288] T 00000000000000000000000000000000 P 726f37f70bde44e699103131c029838d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:31:18.413322 2180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 726f37f70bde44e699103131c029838d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:31:18.414060 2180 raft_consensus.cc:686] T 00000000000000000000000000000000 P 726f37f70bde44e699103131c029838d [term 1 LEADER]: Becoming Leader. State: Replica: 726f37f70bde44e699103131c029838d, State: Running, Role: LEADER
I20260926 03:31:18.414546 2180 consensus_queue.cc:237] T 00000000000000000000000000000000 P 726f37f70bde44e699103131c029838d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "726f37f70bde44e699103131c029838d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33565 } }
I20260926 03:31:18.415269 2175 sys_catalog.cc:564] T 00000000000000000000000000000000 P 726f37f70bde44e699103131c029838d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:31:18.417474 2182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 726f37f70bde44e699103131c029838d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 726f37f70bde44e699103131c029838d. Latest consensus state: current_term: 1 leader_uuid: "726f37f70bde44e699103131c029838d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "726f37f70bde44e699103131c029838d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33565 } } }
I20260926 03:31:18.417879 2182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 726f37f70bde44e699103131c029838d [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:18.418963 2181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 726f37f70bde44e699103131c029838d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "726f37f70bde44e699103131c029838d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "726f37f70bde44e699103131c029838d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33565 } } }
I20260926 03:31:18.419348 2181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 726f37f70bde44e699103131c029838d [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:18.419870 2186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:31:18.425246 2186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:31:18.434361 2186 catalog_manager.cc:1347] Generated new cluster ID: 68526ebeed854cae9742d977c68ef757
I20260926 03:31:18.434525 2186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:31:18.450029 2186 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:31:18.451273 2186 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:31:18.461314 2186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 726f37f70bde44e699103131c029838d: Generated new TSK 0
I20260926 03:31:18.462129 2186 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:31:18.508145 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33565
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:18.829365 2201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:18.829697 2201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:18.829876 2201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:18.836936 2201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:18.837086 2201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:18.837251 2201 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 03:31:18.837338 2201 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:31:18.837476 2201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:31:18.847007 2201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:33565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:18.848258 2201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:18.849632 2201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:18.862205 2207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:18.862179 2206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:18.862130 2209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:18.863039 2201 server_base.cc:1034] running on GCE node
I20260926 03:31:18.863605 2201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:18.864456 2201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:18.865685 2201 hybrid_clock.cc:648] HybridClock initialized: now 1790393478865647 us; error 28 us; skew 500 ppm
I20260926 03:31:18.865988 2201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:18 dist-test-slave-dt7p krb5kdc[2085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393478, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:18.868939 2201 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:31:18.870299 2201 webserver.cc:460] Webserver started at http://127.0.97.1:40093/ using document root <none> and password file <none>
I20260926 03:31:18.870937 2201 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:18.871083 2201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:18.871376 2201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:18.874135 2201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-0/data/instance:
uuid: "7f20b7a1aaba41a6b2c0c190047f7f98"
format_stamp: "Formatted at 2026-09-26 03:31:18 on dist-test-slave-dt7p"
server_key: "f6075f075f0e9f29d37b04dac537afbf"
server_key_iv: "c84b985c660470db27dd94a07bef8b88"
server_key_version: "encryptionkey@0"
I20260926 03:31:18.874876 2201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-0/wal/instance:
uuid: "7f20b7a1aaba41a6b2c0c190047f7f98"
format_stamp: "Formatted at 2026-09-26 03:31:18 on dist-test-slave-dt7p"
server_key: "f6075f075f0e9f29d37b04dac537afbf"
server_key_iv: "c84b985c660470db27dd94a07bef8b88"
server_key_version: "encryptionkey@0"
I20260926 03:31:18.879666 2201 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 03:31:18.882965 2216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:18.884152 2201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:31:18.884320 2201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-0/wal
uuid: "7f20b7a1aaba41a6b2c0c190047f7f98"
format_stamp: "Formatted at 2026-09-26 03:31:18 on dist-test-slave-dt7p"
server_key: "f6075f075f0e9f29d37b04dac537afbf"
server_key_iv: "c84b985c660470db27dd94a07bef8b88"
server_key_version: "encryptionkey@0"
I20260926 03:31:18.884514 2201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:18.913331 2201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:18.917411 2201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:18.917702 2201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:18.920764 2201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:31:18.920890 2201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:18.921065 2201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:31:18.921185 2201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 26 03:31:18 dist-test-slave-dt7p krb5kdc[2085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393478, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:31:18.941334 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:18.921268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57735 (local address 127.0.97.62:33565)
0926 03:31:18.921772 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:18.921782 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:18.922861 (+ 1079us) server_negotiation.cc:407] Connection header received
0926 03:31:18.924785 (+ 1924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:18.924791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:18.924873 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:18.925005 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:18.926631 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:18.927322 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:18.929809 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:18.930225 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:18.930357 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:18.934199 (+ 3842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:18.934208 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:18.934214 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:18.934261 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:18.937492 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:18.938504 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:18.938510 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:18.938514 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:18.938662 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:18.939345 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:18.939351 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:18.939356 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:18.939687 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:18.939844 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:18.940902 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0926 03:31:18.941094 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":232,"threads_started":1}
I20260926 03:31:18.943203 2226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:18.921835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33565 (local address 127.0.97.1:57735)
0926 03:31:18.922707 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:18.922756 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:18.924460 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:18.925285 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:18.925302 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:18.925829 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:18.926447 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:18.926465 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:18.927502 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:18.927507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:18.929633 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:18.929667 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:18.930444 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:18.930450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:18.930673 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:18.931387 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:18.931407 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:18.933952 (+ 2545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:18.937644 (+ 3692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:18.937657 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:18.937676 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:18.938349 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:18.938840 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:18.938843 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:18.938847 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:18.939185 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:18.939846 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:18.939862 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:18.940371 (+ 509us) client_negotiation.cc:769] Sending connection context
0926 03:31:18.940775 (+ 404us) client_negotiation.cc:240] Negotiation successful
0926 03:31:18.941414 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":403,"threads_started":1}
I20260926 03:31:18.962639 2266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:18.949364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33565 (local address 127.0.97.1:48009)
0926 03:31:18.949990 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:18.950004 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:18.950156 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:18.950451 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:18.950455 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:18.950894 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:18.951151 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:18.951159 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:18.952105 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:18.952110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:18.953387 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:18.953401 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:18.954119 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:18.954125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:18.954363 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:18.955095 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:18.955129 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:18.956466 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:18.959798 (+ 3332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:18.959803 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:18.959807 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:18.960304 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:18.960955 (+ 651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:18.960962 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:18.960965 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:18.961094 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:18.961843 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:18.961846 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:18.961989 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 03:31:18.962124 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 03:31:18.962330 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":358,"threads_started":1}
I20260926 03:31:18.963035 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:18.949223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48009 (local address 127.0.97.62:33565)
0926 03:31:18.949371 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:18.949375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:18.950092 (+ 717us) server_negotiation.cc:407] Connection header received
0926 03:31:18.950265 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:18.950270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:18.950334 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:18.950431 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:18.951282 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:18.951968 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:18.953539 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:18.953941 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:18.954074 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:18.956701 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:18.956710 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:18.956715 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:18.956750 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:18.959667 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:18.960638 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:18.960644 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:18.960648 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:18.960800 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:18.961216 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:18.961224 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:18.961230 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:18.961709 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:18.961896 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:18.962266 (+ 370us) server_negotiation.cc:299] Negotiation successful
0926 03:31:18.962545 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 03:31:18.980754 2201 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46239
I20260926 03:31:18.980777 2336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46239 every 8 connection(s)
I20260926 03:31:18.982015 2201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:31:18.984140 2337 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33565
I20260926 03:31:18.984367 2337 heartbeater.cc:464] Registering TS with master...
I20260926 03:31:18.985082 2337 heartbeater.cc:511] Master 127.0.97.62:33565 requested a full tablet report, sending...
I20260926 03:31:18.986454 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 2201
I20260926 03:31:18.986637 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:31:18.987054 2130 ts_manager.cc:194] Registered new tserver with Master: 7f20b7a1aaba41a6b2c0c190047f7f98 (127.0.97.1:46239)
I20260926 03:31:18.987308 388 external_mini_cluster.cc:1428] Setting key dc2d752d7524b503f9512ef0ef1d8595
I20260926 03:31:18.989403 2130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57735
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:31:19.029639 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33565
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:19.340890 2341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:19.341156 2341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:19.341346 2341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:19.348531 2341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:19.348664 2341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:19.348856 2341 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 03:31:19.348935 2341 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:31:19.349071 2341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:31:19.358814 2341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:33565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:19.360199 2341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:19.361653 2341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:19.376861 2347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:19.377295 2346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:19.377315 2349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:19.377964 2341 server_base.cc:1034] running on GCE node
I20260926 03:31:19.378556 2341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:19.379480 2341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:19.380697 2341 hybrid_clock.cc:648] HybridClock initialized: now 1790393479380642 us; error 43 us; skew 500 ppm
I20260926 03:31:19.381004 2341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:19 dist-test-slave-dt7p krb5kdc[2085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:19.384153 2341 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:31:19.385715 2341 webserver.cc:460] Webserver started at http://127.0.97.2:42955/ using document root <none> and password file <none>
I20260926 03:31:19.386377 2341 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:19.386499 2341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:19.386817 2341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:19.389963 2341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-1/data/instance:
uuid: "7ef0f75e0d754f0597ef751b629233dc"
format_stamp: "Formatted at 2026-09-26 03:31:19 on dist-test-slave-dt7p"
server_key: "b35a2dbbdb95f28fa37b0e4be2b1b8d0"
server_key_iv: "a55040b0c610bf86a92a30f86430cbc5"
server_key_version: "encryptionkey@0"
I20260926 03:31:19.390714 2341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7ef0f75e0d754f0597ef751b629233dc"
format_stamp: "Formatted at 2026-09-26 03:31:19 on dist-test-slave-dt7p"
server_key: "b35a2dbbdb95f28fa37b0e4be2b1b8d0"
server_key_iv: "a55040b0c610bf86a92a30f86430cbc5"
server_key_version: "encryptionkey@0"
I20260926 03:31:19.395833 2341 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:31:19.399219 2356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:19.400426 2341 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:31:19.400656 2341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-1/wal
uuid: "7ef0f75e0d754f0597ef751b629233dc"
format_stamp: "Formatted at 2026-09-26 03:31:19 on dist-test-slave-dt7p"
server_key: "b35a2dbbdb95f28fa37b0e4be2b1b8d0"
server_key_iv: "a55040b0c610bf86a92a30f86430cbc5"
server_key_version: "encryptionkey@0"
I20260926 03:31:19.400844 2341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:19.434986 2341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:19.439242 2341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:19.439527 2341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:19.442200 2341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:31:19.442345 2341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:19.442479 2341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:31:19.442637 2341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 26 03:31:19 dist-test-slave-dt7p krb5kdc[2085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:31:19.468901 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:19.442685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47809 (local address 127.0.97.62:33565)
0926 03:31:19.442875 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:19.442882 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:19.444301 (+ 1419us) server_negotiation.cc:407] Connection header received
0926 03:31:19.446696 (+ 2395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:19.446705 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:19.446790 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:19.446900 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:19.449072 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:19.450176 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:19.452747 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:19.453558 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:19.453716 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:19.458688 (+ 4972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:19.458698 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:19.458706 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:19.458752 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:19.464015 (+ 5263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:19.465303 (+ 1288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:19.465308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:19.465312 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:19.465467 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:19.466366 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:19.466371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:19.466374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:19.466778 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:19.466955 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:19.468121 (+ 1166us) server_negotiation.cc:299] Negotiation successful
0926 03:31:19.468392 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 03:31:19.470664 2365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:19.443276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33565 (local address 127.0.97.2:47809)
0926 03:31:19.444128 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:19.444178 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:19.446367 (+ 2189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:19.447146 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:19.447157 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:19.447860 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:19.448817 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:19.448850 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:19.450405 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:19.450418 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:19.452541 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:19.452552 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:19.453942 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:19.453951 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:19.454149 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:19.455006 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:19.455031 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:19.458391 (+ 3360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:19.464206 (+ 5815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:19.464218 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:19.464242 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:19.465149 (+ 907us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:19.465674 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:19.465685 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:19.465723 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:19.466182 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:19.466963 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:19.466971 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:19.467547 (+ 576us) client_negotiation.cc:769] Sending connection context
0926 03:31:19.467977 (+ 430us) client_negotiation.cc:240] Negotiation successful
0926 03:31:19.468714 (+ 737us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":408,"threads_started":1}
I20260926 03:31:19.481702 2341 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41135
I20260926 03:31:19.481748 2475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41135 every 8 connection(s)
I20260926 03:31:19.482947 2341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:31:19.484939 2476 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33565
I20260926 03:31:19.485178 2476 heartbeater.cc:464] Registering TS with master...
I20260926 03:31:19.485865 2476 heartbeater.cc:511] Master 127.0.97.62:33565 requested a full tablet report, sending...
I20260926 03:31:19.487007 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 2341
I20260926 03:31:19.487185 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:31:19.487653 388 external_mini_cluster.cc:1428] Setting key 99700791f1bfd8a589512461c89b92fa
I20260926 03:31:19.488910 2130 ts_manager.cc:194] Registered new tserver with Master: 7ef0f75e0d754f0597ef751b629233dc (127.0.97.2:41135)
I20260926 03:31:19.489893 2130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47809
I20260926 03:31:19.493276 2471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:19.479084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33565 (local address 127.0.97.2:47947)
0926 03:31:19.479965 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:19.479980 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:19.480143 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:19.480552 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:19.480557 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:19.481069 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:19.481517 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:19.481531 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:19.482615 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:19.482623 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:19.484007 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:19.484022 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:19.485211 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:19.485215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:19.485419 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:19.486055 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:19.486072 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:19.486875 (+ 803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:19.490645 (+ 3770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:19.490649 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:19.490653 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:19.491156 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:19.491586 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:19.491591 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:19.491594 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:19.491729 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:19.492417 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:19.492421 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:19.492634 (+ 213us) client_negotiation.cc:769] Sending connection context
0926 03:31:19.492772 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 03:31:19.492977 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":624,"threads_started":1}
I20260926 03:31:19.493539 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:19.479493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47947 (local address 127.0.97.62:33565)
0926 03:31:19.479719 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:19.479727 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:19.480062 (+ 335us) server_negotiation.cc:407] Connection header received
0926 03:31:19.480333 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:19.480338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:19.480412 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:19.480519 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:19.481668 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:19.482479 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:19.484189 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:19.484813 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:19.484925 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:19.487044 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:19.487050 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:19.487055 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:19.487094 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:19.490511 (+ 3417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:19.491295 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:19.491303 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:19.491309 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:19.491455 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:19.491850 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:19.491858 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:19.491863 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:19.492284 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:19.492435 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:19.492892 (+ 457us) server_negotiation.cc:299] Negotiation successful
0926 03:31:19.493127 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:31:19.528371 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33565
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:19.847378 2480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:19.847661 2480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:19.847759 2480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:19.854857 2480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:19.854956 2480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:19.855036 2480 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 03:31:19.855087 2480 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:31:19.855147 2480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:31:19.865032 2480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:33565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:19.866310 2480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:19.867695 2480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:19.881449 2485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:19.881443 2488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:19.881484 2486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:19.882242 2480 server_base.cc:1034] running on GCE node
I20260926 03:31:19.882939 2480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:19.883841 2480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:19.885049 2480 hybrid_clock.cc:648] HybridClock initialized: now 1790393479884977 us; error 65 us; skew 500 ppm
I20260926 03:31:19.885394 2480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:19 dist-test-slave-dt7p krb5kdc[2085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:19.888742 2480 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:31:19.890377 2480 webserver.cc:460] Webserver started at http://127.0.97.3:38071/ using document root <none> and password file <none>
I20260926 03:31:19.891139 2480 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:19.891297 2480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:19.891625 2480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:19.894835 2480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-2/data/instance:
uuid: "9ca20d03820c40e6a7fe2efadf6ef182"
format_stamp: "Formatted at 2026-09-26 03:31:19 on dist-test-slave-dt7p"
server_key: "9324ae0f7f4424c142c750585737b2c5"
server_key_iv: "bb6320faf82a2a11bf041172897e3efd"
server_key_version: "encryptionkey@0"
I20260926 03:31:19.895691 2480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9ca20d03820c40e6a7fe2efadf6ef182"
format_stamp: "Formatted at 2026-09-26 03:31:19 on dist-test-slave-dt7p"
server_key: "9324ae0f7f4424c142c750585737b2c5"
server_key_iv: "bb6320faf82a2a11bf041172897e3efd"
server_key_version: "encryptionkey@0"
I20260926 03:31:19.902575 2480 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260926 03:31:19.907541 2495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:19.909006 2480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:31:19.909210 2480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-2/wal
uuid: "9ca20d03820c40e6a7fe2efadf6ef182"
format_stamp: "Formatted at 2026-09-26 03:31:19 on dist-test-slave-dt7p"
server_key: "9324ae0f7f4424c142c750585737b2c5"
server_key_iv: "bb6320faf82a2a11bf041172897e3efd"
server_key_version: "encryptionkey@0"
I20260926 03:31:19.909421 2480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:19.931984 2480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:19.936431 2480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:19.936770 2480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:19.939906 2480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:31:19.940032 2480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:19.940153 2480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:31:19.940258 2480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:31:19 dist-test-slave-dt7p krb5kdc[2085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:31:19.963090 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:19.940399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58435 (local address 127.0.97.62:33565)
0926 03:31:19.940631 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:19.940639 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:19.942142 (+ 1503us) server_negotiation.cc:407] Connection header received
0926 03:31:19.944259 (+ 2117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:19.944265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:19.944351 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:19.944479 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:19.946899 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:19.947964 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:19.950556 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:19.950954 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:19.951085 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:19.955825 (+ 4740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:19.955831 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:19.955834 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:19.955865 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:19.959454 (+ 3589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:19.960548 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:19.960556 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:19.960585 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:19.960770 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:19.961284 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:19.961290 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:19.961293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:19.961601 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:19.961755 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:19.962504 (+ 749us) server_negotiation.cc:299] Negotiation successful
0926 03:31:19.962730 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 03:31:19.964200 2504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:19.941152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33565 (local address 127.0.97.3:58435)
0926 03:31:19.941982 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:19.942025 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:19.943963 (+ 1938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:19.944784 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:19.944801 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:19.945702 (+ 901us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:19.946654 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:19.946687 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:19.948162 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:19.948165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:19.950354 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:19.950372 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:19.951434 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:19.951439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:19.951757 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:19.952660 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:19.952703 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:19.955533 (+ 2830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:19.959607 (+ 4074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:19.959628 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:19.959643 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:19.960370 (+ 727us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:19.960905 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:19.960911 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:19.960923 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:19.961145 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:19.961741 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:19.961749 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:19.962127 (+ 378us) client_negotiation.cc:769] Sending connection context
0926 03:31:19.962397 (+ 270us) client_negotiation.cc:240] Negotiation successful
0926 03:31:19.962950 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":359,"threads_started":1}
I20260926 03:31:19.987993 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:19.971267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54115 (local address 127.0.97.62:33565)
0926 03:31:19.971448 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:19.971456 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:19.972404 (+ 948us) server_negotiation.cc:407] Connection header received
0926 03:31:19.972671 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:19.972680 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:19.972797 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:19.972914 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:19.974885 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:19.975596 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:19.978207 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:19.978659 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:19.978803 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:19.980919 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:19.980925 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:19.980930 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:19.980965 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:19.984708 (+ 3743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:19.985592 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:19.985598 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:19.985602 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:19.985745 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:19.986310 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:19.986315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:19.986317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:19.986662 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:19.986777 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:19.987401 (+ 624us) server_negotiation.cc:299] Negotiation successful
0926 03:31:19.987644 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 03:31:19.989346 2575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:19.971317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33565 (local address 127.0.97.3:54115)
0926 03:31:19.972258 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:19.972285 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:19.972459 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:19.973084 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:19.973088 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:19.974152 (+ 1064us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:19.974706 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:19.974721 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:19.975739 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:19.975747 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:19.977956 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:19.977974 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:19.978848 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:19.978855 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:19.979010 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:19.979644 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:19.979662 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:19.980711 (+ 1049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:19.984865 (+ 4154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:19.984870 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:19.984875 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:19.985436 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:19.985900 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:19.985904 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:19.985907 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:19.986171 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:19.986875 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:19.986880 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:19.987084 (+ 204us) client_negotiation.cc:769] Sending connection context
0926 03:31:19.987922 (+ 838us) client_negotiation.cc:240] Negotiation successful
0926 03:31:19.988732 (+ 810us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"thread_start_us":442,"threads_started":1}
I20260926 03:31:19.992125 2337 heartbeater.cc:503] Master 127.0.97.62:33565 was elected leader, sending a full tablet report...
I20260926 03:31:19.996101 2480 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41599
I20260926 03:31:19.996137 2614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41599 every 8 connection(s)
I20260926 03:31:19.997423 2480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:31:19.999482 2615 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33565
I20260926 03:31:19.999665 2615 heartbeater.cc:464] Registering TS with master...
I20260926 03:31:20.000239 2615 heartbeater.cc:511] Master 127.0.97.62:33565 requested a full tablet report, sending...
I20260926 03:31:20.001480 2130 ts_manager.cc:194] Registered new tserver with Master: 9ca20d03820c40e6a7fe2efadf6ef182 (127.0.97.3:41599)
I20260926 03:31:20.002269 2130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:58435
I20260926 03:31:20.006784 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 2480
I20260926 03:31:20.006922 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:31:20.007368 388 external_mini_cluster.cc:1428] Setting key b90e8425556e0eeb68ed7a727d1d98ef
I20260926 03:31:20.010351 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 03:31:20 dist-test-slave-dt7p krb5kdc[2085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 03:31:20 dist-test-slave-dt7p krb5kdc[2085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:31:20.058722 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:20.044661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53038 (local address 127.0.97.62:33565)
0926 03:31:20.044846 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:20.044855 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:20.044993 (+ 138us) server_negotiation.cc:407] Connection header received
0926 03:31:20.045157 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:20.045161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:20.045241 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:20.045369 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:20.046849 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.047570 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.048875 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.049304 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.049438 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:20.053293 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:20.053299 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:20.053302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:20.053335 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:20.055725 (+ 2390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:20.056367 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:20.056372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:20.056374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:20.056491 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:20.056933 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:20.056937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:20.056939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:20.057197 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:20.057397 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:20.058202 (+ 805us) server_negotiation.cc:299] Negotiation successful
0926 03:31:20.058421 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 03:31:20.062762 2130 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:31:20.070194 2130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:31:20.097754 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:20.083816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41135 (local address 127.0.0.1:58674)
0926 03:31:20.085794 (+ 1978us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:20.085811 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:20.085926 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:20.087456 (+ 1530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:20.087467 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:20.087506 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:31:20.088005 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:20.088021 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.090624 (+ 2603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.090632 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:20.093917 (+ 3285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:20.093938 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.096700 (+ 2762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.096707 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:20.096912 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:20.096985 (+ 73us) client_negotiation.cc:769] Sending connection context
0926 03:31:20.097139 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 03:31:20.097348 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1847,"thread_start_us":583,"threads_started":1}
I20260926 03:31:20.098814 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:20.084139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58674 (local address 127.0.97.2:41135)
0926 03:31:20.086472 (+ 2333us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:20.086482 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:20.086518 (+ 36us) server_negotiation.cc:407] Connection header received
0926 03:31:20.086649 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:20.086660 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:20.086984 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:20.087200 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:31:20.088322 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.090380 (+ 2058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.094201 (+ 3821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.096140 (+ 1939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.096285 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:20.096552 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:20.098327 (+ 1775us) server_negotiation.cc:299] Negotiation successful
0926 03:31:20.098510 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2127,"thread_start_us":2009,"threads_started":1}
I20260926 03:31:20.102942 2406 tablet_service.cc:1467] Processing CreateTablet for tablet ce157a73fb324adfa46a1864fd75297c (DEFAULT_TABLE table=test-table [id=d325ef6e81ad498190f0f5f7151a592c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:31:20.104477 2406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce157a73fb324adfa46a1864fd75297c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:20.105281 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:20.085398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41599 (local address 127.0.0.1:39320)
0926 03:31:20.097290 (+ 11892us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:20.097307 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:20.097414 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:20.098387 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:20.098391 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:20.098414 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:31:20.098740 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:20.098750 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.100766 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.100771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:20.102369 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:20.102380 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.104365 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.104373 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:20.104552 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:20.104610 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 03:31:20.104699 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 03:31:20.104823 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11778,"mutex_wait_us":1,"thread_start_us":11608,"threads_started":1}
I20260926 03:31:20.106098 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:20.083589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46239 (local address 127.0.0.1:55762)
0926 03:31:20.084982 (+ 1393us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:20.085038 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:20.085251 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:20.097555 (+ 12304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:20.097560 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:20.097585 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:31:20.097961 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:20.097975 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.101264 (+ 3289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.101268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:20.103273 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:20.103286 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.105365 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.105371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:20.105559 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:20.105593 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 03:31:20.105674 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 03:31:20.105797 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1026,"thread_start_us":2179,"threads_started":1}
I20260926 03:31:20.106150 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:20.085625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39320 (local address 127.0.97.3:41599)
0926 03:31:20.096690 (+ 11065us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:20.096699 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:20.097539 (+ 840us) server_negotiation.cc:407] Connection header received
0926 03:31:20.097635 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:20.097643 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:20.097895 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:20.098096 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:31:20.099026 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.100590 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.102538 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.103542 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.103666 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:20.103892 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:20.105665 (+ 1773us) server_negotiation.cc:299] Negotiation successful
0926 03:31:20.105825 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10860,"thread_start_us":11076,"threads_started":1}
I20260926 03:31:20.106451 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:20.083754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55762 (local address 127.0.97.1:46239)
0926 03:31:20.096612 (+ 12858us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:20.096626 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:20.096669 (+ 43us) server_negotiation.cc:407] Connection header received
0926 03:31:20.096796 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:20.096807 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:20.097134 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:20.098180 (+ 1046us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:31:20.098334 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.101084 (+ 2750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.103485 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.105152 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.105291 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:20.105735 (+ 444us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:20.105796 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 03:31:20.105940 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12613,"thread_start_us":13534,"threads_started":1}
I20260926 03:31:20.109398 2545 tablet_service.cc:1467] Processing CreateTablet for tablet ce157a73fb324adfa46a1864fd75297c (DEFAULT_TABLE table=test-table [id=d325ef6e81ad498190f0f5f7151a592c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:31:20.109716 2272 tablet_service.cc:1467] Processing CreateTablet for tablet ce157a73fb324adfa46a1864fd75297c (DEFAULT_TABLE table=test-table [id=d325ef6e81ad498190f0f5f7151a592c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:31:20.110759 2545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce157a73fb324adfa46a1864fd75297c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:20.110962 2272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce157a73fb324adfa46a1864fd75297c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:20.119838 2634 tablet_bootstrap.cc:492] T ce157a73fb324adfa46a1864fd75297c P 7ef0f75e0d754f0597ef751b629233dc: Bootstrap starting.
I20260926 03:31:20.122850 2635 tablet_bootstrap.cc:492] T ce157a73fb324adfa46a1864fd75297c P 9ca20d03820c40e6a7fe2efadf6ef182: Bootstrap starting.
I20260926 03:31:20.123920 2634 tablet_bootstrap.cc:654] T ce157a73fb324adfa46a1864fd75297c P 7ef0f75e0d754f0597ef751b629233dc: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:20.124620 2636 tablet_bootstrap.cc:492] T ce157a73fb324adfa46a1864fd75297c P 7f20b7a1aaba41a6b2c0c190047f7f98: Bootstrap starting.
I20260926 03:31:20.125801 2634 log.cc:824] T ce157a73fb324adfa46a1864fd75297c P 7ef0f75e0d754f0597ef751b629233dc: Log is configured to *not* fsync() on all Append() calls
I20260926 03:31:20.125989 2635 tablet_bootstrap.cc:654] T ce157a73fb324adfa46a1864fd75297c P 9ca20d03820c40e6a7fe2efadf6ef182: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:20.127074 2635 log.cc:824] T ce157a73fb324adfa46a1864fd75297c P 9ca20d03820c40e6a7fe2efadf6ef182: Log is configured to *not* fsync() on all Append() calls
I20260926 03:31:20.127784 2636 tablet_bootstrap.cc:654] T ce157a73fb324adfa46a1864fd75297c P 7f20b7a1aaba41a6b2c0c190047f7f98: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:20.128952 2636 log.cc:824] T ce157a73fb324adfa46a1864fd75297c P 7f20b7a1aaba41a6b2c0c190047f7f98: Log is configured to *not* fsync() on all Append() calls
I20260926 03:31:20.129163 2634 tablet_bootstrap.cc:492] T ce157a73fb324adfa46a1864fd75297c P 7ef0f75e0d754f0597ef751b629233dc: No bootstrap required, opened a new log
I20260926 03:31:20.129451 2634 ts_tablet_manager.cc:1397] T ce157a73fb324adfa46a1864fd75297c P 7ef0f75e0d754f0597ef751b629233dc: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 03:31:20.130554 2635 tablet_bootstrap.cc:492] T ce157a73fb324adfa46a1864fd75297c P 9ca20d03820c40e6a7fe2efadf6ef182: No bootstrap required, opened a new log
I20260926 03:31:20.130824 2635 ts_tablet_manager.cc:1397] T ce157a73fb324adfa46a1864fd75297c P 9ca20d03820c40e6a7fe2efadf6ef182: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 03:31:20.131881 2636 tablet_bootstrap.cc:492] T ce157a73fb324adfa46a1864fd75297c P 7f20b7a1aaba41a6b2c0c190047f7f98: No bootstrap required, opened a new log
I20260926 03:31:20.132129 2636 ts_tablet_manager.cc:1397] T ce157a73fb324adfa46a1864fd75297c P 7f20b7a1aaba41a6b2c0c190047f7f98: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260926 03:31:20.138418 2635 raft_consensus.cc:348] T ce157a73fb324adfa46a1864fd75297c P 9ca20d03820c40e6a7fe2efadf6ef182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:20.138790 2635 raft_consensus.cc:374] T ce157a73fb324adfa46a1864fd75297c P 9ca20d03820c40e6a7fe2efadf6ef182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:20.138880 2635 raft_consensus.cc:729] T ce157a73fb324adfa46a1864fd75297c P 9ca20d03820c40e6a7fe2efadf6ef182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ca20d03820c40e6a7fe2efadf6ef182, State: Initialized, Role: FOLLOWER
I20260926 03:31:20.139529 2635 consensus_queue.cc:260] T ce157a73fb324adfa46a1864fd75297c P 9ca20d03820c40e6a7fe2efadf6ef182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:20.139806 2634 raft_consensus.cc:348] T ce157a73fb324adfa46a1864fd75297c P 7ef0f75e0d754f0597ef751b629233dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:20.140202 2634 raft_consensus.cc:374] T ce157a73fb324adfa46a1864fd75297c P 7ef0f75e0d754f0597ef751b629233dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:20.140298 2634 raft_consensus.cc:729] T ce157a73fb324adfa46a1864fd75297c P 7ef0f75e0d754f0597ef751b629233dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef0f75e0d754f0597ef751b629233dc, State: Initialized, Role: FOLLOWER
I20260926 03:31:20.140722 2615 heartbeater.cc:503] Master 127.0.97.62:33565 was elected leader, sending a full tablet report...
I20260926 03:31:20.140990 2634 consensus_queue.cc:260] T ce157a73fb324adfa46a1864fd75297c P 7ef0f75e0d754f0597ef751b629233dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:20.142060 2476 heartbeater.cc:503] Master 127.0.97.62:33565 was elected leader, sending a full tablet report...
I20260926 03:31:20.142409 2636 raft_consensus.cc:348] T ce157a73fb324adfa46a1864fd75297c P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:20.142812 2634 ts_tablet_manager.cc:1428] T ce157a73fb324adfa46a1864fd75297c P 7ef0f75e0d754f0597ef751b629233dc: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260926 03:31:20.142805 2636 raft_consensus.cc:374] T ce157a73fb324adfa46a1864fd75297c P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:20.142894 2636 raft_consensus.cc:729] T ce157a73fb324adfa46a1864fd75297c P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f20b7a1aaba41a6b2c0c190047f7f98, State: Initialized, Role: FOLLOWER
I20260926 03:31:20.143518 2636 consensus_queue.cc:260] T ce157a73fb324adfa46a1864fd75297c P 7f20b7a1aaba41a6b2c0c190047f7f98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:20.144309 2635 ts_tablet_manager.cc:1428] T ce157a73fb324adfa46a1864fd75297c P 9ca20d03820c40e6a7fe2efadf6ef182: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260926 03:31:20.145905 2636 ts_tablet_manager.cc:1428] T ce157a73fb324adfa46a1864fd75297c P 7f20b7a1aaba41a6b2c0c190047f7f98: Time spent starting tablet: real 0.014s user 0.012s sys 0.003s
W20260926 03:31:20.234447 2338 tablet.cc:2369] T ce157a73fb324adfa46a1864fd75297c P 7f20b7a1aaba41a6b2c0c190047f7f98: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:31:20.234948 2477 tablet.cc:2369] T ce157a73fb324adfa46a1864fd75297c P 7ef0f75e0d754f0597ef751b629233dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:31:20.248967 2616 tablet.cc:2369] T ce157a73fb324adfa46a1864fd75297c P 9ca20d03820c40e6a7fe2efadf6ef182: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:31:20.512413 2641 raft_consensus.cc:482] T ce157a73fb324adfa46a1864fd75297c P 7ef0f75e0d754f0597ef751b629233dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:31:20.512823 2641 raft_consensus.cc:504] T ce157a73fb324adfa46a1864fd75297c P 7ef0f75e0d754f0597ef751b629233dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:20.514891 2641 leader_election.cc:288] T ce157a73fb324adfa46a1864fd75297c P 7ef0f75e0d754f0597ef751b629233dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f20b7a1aaba41a6b2c0c190047f7f98 (127.0.97.1:46239), 9ca20d03820c40e6a7fe2efadf6ef182 (127.0.97.3:41599)
I20260926 03:31:20.523438 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:20.515402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46527 (local address 127.0.97.1:46239)
0926 03:31:20.515830 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:20.515836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:20.516358 (+ 522us) server_negotiation.cc:407] Connection header received
0926 03:31:20.516638 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:20.516645 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:20.516733 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:20.516812 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:31:20.517408 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.518517 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.521279 (+ 2762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.522426 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.522586 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:20.522702 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:20.523139 (+ 437us) server_negotiation.cc:299] Negotiation successful
0926 03:31:20.523233 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293}
I20260926 03:31:20.523836 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:20.515421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57745 (local address 127.0.97.3:41599)
0926 03:31:20.515659 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:20.515668 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:20.516743 (+ 1075us) server_negotiation.cc:407] Connection header received
0926 03:31:20.516947 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:20.516953 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:20.517019 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:20.517123 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:31:20.517735 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.518713 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.521605 (+ 2892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.522877 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.522972 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:20.523042 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:20.523489 (+ 447us) server_negotiation.cc:299] Negotiation successful
0926 03:31:20.523626 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 03:31:20.523931 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:20.515205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41599 (local address 127.0.97.2:57745)
0926 03:31:20.516639 (+ 1434us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:20.516657 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:20.516803 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:20.517181 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:20.517186 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:20.517213 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:31:20.517585 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:20.517593 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.518843 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.518848 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:20.521446 (+ 2598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:20.521456 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.523143 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.523149 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:20.523315 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:20.523357 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 03:31:20.523447 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 03:31:20.523571 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"mutex_wait_us":40,"thread_start_us":754,"threads_started":1}
I20260926 03:31:20.523952 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:20.515253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46239 (local address 127.0.97.2:46527)
0926 03:31:20.516211 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:20.516230 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:20.516393 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:20.516833 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:20.516840 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:20.516880 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:31:20.517214 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:20.517229 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.518646 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.518650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:20.521071 (+ 2421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:20.521085 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.522777 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.522785 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:20.522912 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:20.522972 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 03:31:20.523086 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 03:31:20.523235 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"mutex_wait_us":63,"thread_start_us":400,"threads_started":1}
I20260926 03:31:20.525166 2569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce157a73fb324adfa46a1864fd75297c" candidate_uuid: "7ef0f75e0d754f0597ef751b629233dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" is_pre_election: true
I20260926 03:31:20.525182 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce157a73fb324adfa46a1864fd75297c" candidate_uuid: "7ef0f75e0d754f0597ef751b629233dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" is_pre_election: true
I20260926 03:31:20.525715 2569 raft_consensus.cc:2454] T ce157a73fb324adfa46a1864fd75297c P 9ca20d03820c40e6a7fe2efadf6ef182 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ef0f75e0d754f0597ef751b629233dc in term 0.
I20260926 03:31:20.525715 2292 raft_consensus.cc:2454] T ce157a73fb324adfa46a1864fd75297c P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ef0f75e0d754f0597ef751b629233dc in term 0.
I20260926 03:31:20.526396 2357 leader_election.cc:302] T ce157a73fb324adfa46a1864fd75297c P 7ef0f75e0d754f0597ef751b629233dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ef0f75e0d754f0597ef751b629233dc, 7f20b7a1aaba41a6b2c0c190047f7f98; no voters:
I20260926 03:31:20.526743 2641 raft_consensus.cc:2781] T ce157a73fb324adfa46a1864fd75297c P 7ef0f75e0d754f0597ef751b629233dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:31:20.526942 2641 raft_consensus.cc:482] T ce157a73fb324adfa46a1864fd75297c P 7ef0f75e0d754f0597ef751b629233dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:31:20.527055 2641 raft_consensus.cc:3037] T ce157a73fb324adfa46a1864fd75297c P 7ef0f75e0d754f0597ef751b629233dc [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:20.529337 2641 raft_consensus.cc:504] T ce157a73fb324adfa46a1864fd75297c P 7ef0f75e0d754f0597ef751b629233dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:20.529901 2641 leader_election.cc:288] T ce157a73fb324adfa46a1864fd75297c P 7ef0f75e0d754f0597ef751b629233dc [CANDIDATE]: Term 1 election: Requested vote from peers 7f20b7a1aaba41a6b2c0c190047f7f98 (127.0.97.1:46239), 9ca20d03820c40e6a7fe2efadf6ef182 (127.0.97.3:41599)
I20260926 03:31:20.530478 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce157a73fb324adfa46a1864fd75297c" candidate_uuid: "7ef0f75e0d754f0597ef751b629233dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98"
I20260926 03:31:20.530692 2292 raft_consensus.cc:3037] T ce157a73fb324adfa46a1864fd75297c P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:20.530799 2569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce157a73fb324adfa46a1864fd75297c" candidate_uuid: "7ef0f75e0d754f0597ef751b629233dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca20d03820c40e6a7fe2efadf6ef182"
I20260926 03:31:20.531031 2569 raft_consensus.cc:3037] T ce157a73fb324adfa46a1864fd75297c P 9ca20d03820c40e6a7fe2efadf6ef182 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:20.590137 2292 raft_consensus.cc:2454] T ce157a73fb324adfa46a1864fd75297c P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ef0f75e0d754f0597ef751b629233dc in term 1.
I20260926 03:31:20.590148 2569 raft_consensus.cc:2454] T ce157a73fb324adfa46a1864fd75297c P 9ca20d03820c40e6a7fe2efadf6ef182 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ef0f75e0d754f0597ef751b629233dc in term 1.
I20260926 03:31:20.590752 2357 leader_election.cc:302] T ce157a73fb324adfa46a1864fd75297c P 7ef0f75e0d754f0597ef751b629233dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ef0f75e0d754f0597ef751b629233dc, 7f20b7a1aaba41a6b2c0c190047f7f98; no voters:
I20260926 03:31:20.591106 2641 raft_consensus.cc:2781] T ce157a73fb324adfa46a1864fd75297c P 7ef0f75e0d754f0597ef751b629233dc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:31:20.591598 2641 raft_consensus.cc:686] T ce157a73fb324adfa46a1864fd75297c P 7ef0f75e0d754f0597ef751b629233dc [term 1 LEADER]: Becoming Leader. State: Replica: 7ef0f75e0d754f0597ef751b629233dc, State: Running, Role: LEADER
I20260926 03:31:20.592083 2641 consensus_queue.cc:237] T ce157a73fb324adfa46a1864fd75297c P 7ef0f75e0d754f0597ef751b629233dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:20.596529 2128 catalog_manager.cc:5495] T ce157a73fb324adfa46a1864fd75297c P 7ef0f75e0d754f0597ef751b629233dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ef0f75e0d754f0597ef751b629233dc (127.0.97.2). New cstate: current_term: 1 leader_uuid: "7ef0f75e0d754f0597ef751b629233dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } health_report { overall_health: UNKNOWN } } }
Sep 26 03:31:20 dist-test-slave-dt7p krb5kdc[2085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393478, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:31:20.709019 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:20.694946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33565 (local address 127.0.0.1:53042)
0926 03:31:20.695751 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:20.695768 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:20.695904 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:20.696244 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:20.696248 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:20.697048 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:20.697299 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:20.697308 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.698308 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.698312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:20.699380 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:20.699391 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.700083 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.700088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:20.700228 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:20.700943 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:20.700965 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:20.703731 (+ 2766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:20.706765 (+ 3034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:20.706775 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:20.706781 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:20.707208 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:20.707520 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:20.707523 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:20.707526 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:20.707661 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:20.708086 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:20.708092 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:20.708230 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 03:31:20.708371 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 03:31:20.708646 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":490,"threads_started":1}
I20260926 03:31:20.709036 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:20.694920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53042 (local address 127.0.97.62:33565)
0926 03:31:20.695543 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:20.695550 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:20.695853 (+ 303us) server_negotiation.cc:407] Connection header received
0926 03:31:20.696012 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:20.696015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:20.696087 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:20.696187 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:20.697458 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.698183 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.699512 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.699925 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.700017 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:20.703904 (+ 3887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:20.703909 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:20.703912 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:20.703964 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:20.706639 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:20.707309 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:20.707312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:20.707314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:20.707426 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:20.707751 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:20.707754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:20.707756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:20.707979 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:20.708085 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:20.708442 (+ 357us) server_negotiation.cc:299] Negotiation successful
0926 03:31:20.708729 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":136,"server-negotiator.queue_time_us":505,"thread_start_us":343,"threads_started":1}
I20260926 03:31:20.714293 2128 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53042:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 03:31:20.715597 2128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:31:20.722723 2406 tablet_service.cc:1467] Processing CreateTablet for tablet 8f78ad536b9b4e92afd1995e30164f7e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=52a5b0043a314c159bee94dc23afa5dc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:31:20.722494 2272 tablet_service.cc:1467] Processing CreateTablet for tablet 8f78ad536b9b4e92afd1995e30164f7e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=52a5b0043a314c159bee94dc23afa5dc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:31:20.723469 2272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f78ad536b9b4e92afd1995e30164f7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:20.723467 2406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f78ad536b9b4e92afd1995e30164f7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:20.722872 2545 tablet_service.cc:1467] Processing CreateTablet for tablet 8f78ad536b9b4e92afd1995e30164f7e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=52a5b0043a314c159bee94dc23afa5dc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:31:20.724778 2545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f78ad536b9b4e92afd1995e30164f7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:20.730386 2657 tablet_bootstrap.cc:492] T 8f78ad536b9b4e92afd1995e30164f7e P 9ca20d03820c40e6a7fe2efadf6ef182: Bootstrap starting.
I20260926 03:31:20.732218 2657 tablet_bootstrap.cc:654] T 8f78ad536b9b4e92afd1995e30164f7e P 9ca20d03820c40e6a7fe2efadf6ef182: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:20.733085 2658 tablet_bootstrap.cc:492] T 8f78ad536b9b4e92afd1995e30164f7e P 7ef0f75e0d754f0597ef751b629233dc: Bootstrap starting.
I20260926 03:31:20.733960 2659 tablet_bootstrap.cc:492] T 8f78ad536b9b4e92afd1995e30164f7e P 7f20b7a1aaba41a6b2c0c190047f7f98: Bootstrap starting.
I20260926 03:31:20.735570 2659 tablet_bootstrap.cc:654] T 8f78ad536b9b4e92afd1995e30164f7e P 7f20b7a1aaba41a6b2c0c190047f7f98: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:20.735654 2658 tablet_bootstrap.cc:654] T 8f78ad536b9b4e92afd1995e30164f7e P 7ef0f75e0d754f0597ef751b629233dc: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:20.737219 2657 tablet_bootstrap.cc:492] T 8f78ad536b9b4e92afd1995e30164f7e P 9ca20d03820c40e6a7fe2efadf6ef182: No bootstrap required, opened a new log
I20260926 03:31:20.737377 2657 ts_tablet_manager.cc:1397] T 8f78ad536b9b4e92afd1995e30164f7e P 9ca20d03820c40e6a7fe2efadf6ef182: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20260926 03:31:20.737805 2658 tablet_bootstrap.cc:492] T 8f78ad536b9b4e92afd1995e30164f7e P 7ef0f75e0d754f0597ef751b629233dc: No bootstrap required, opened a new log
I20260926 03:31:20.737955 2658 ts_tablet_manager.cc:1397] T 8f78ad536b9b4e92afd1995e30164f7e P 7ef0f75e0d754f0597ef751b629233dc: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260926 03:31:20.738221 2657 raft_consensus.cc:348] T 8f78ad536b9b4e92afd1995e30164f7e P 9ca20d03820c40e6a7fe2efadf6ef182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:20.738464 2657 raft_consensus.cc:374] T 8f78ad536b9b4e92afd1995e30164f7e P 9ca20d03820c40e6a7fe2efadf6ef182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:20.738564 2657 raft_consensus.cc:729] T 8f78ad536b9b4e92afd1995e30164f7e P 9ca20d03820c40e6a7fe2efadf6ef182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ca20d03820c40e6a7fe2efadf6ef182, State: Initialized, Role: FOLLOWER
I20260926 03:31:20.738719 2658 raft_consensus.cc:348] T 8f78ad536b9b4e92afd1995e30164f7e P 7ef0f75e0d754f0597ef751b629233dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:20.738938 2658 raft_consensus.cc:374] T 8f78ad536b9b4e92afd1995e30164f7e P 7ef0f75e0d754f0597ef751b629233dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:20.738793 2657 consensus_queue.cc:260] T 8f78ad536b9b4e92afd1995e30164f7e P 9ca20d03820c40e6a7fe2efadf6ef182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:20.739046 2658 raft_consensus.cc:729] T 8f78ad536b9b4e92afd1995e30164f7e P 7ef0f75e0d754f0597ef751b629233dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef0f75e0d754f0597ef751b629233dc, State: Initialized, Role: FOLLOWER
I20260926 03:31:20.739284 2658 consensus_queue.cc:260] T 8f78ad536b9b4e92afd1995e30164f7e P 7ef0f75e0d754f0597ef751b629233dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:20.739867 2658 ts_tablet_manager.cc:1428] T 8f78ad536b9b4e92afd1995e30164f7e P 7ef0f75e0d754f0597ef751b629233dc: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 03:31:20.740222 2659 tablet_bootstrap.cc:492] T 8f78ad536b9b4e92afd1995e30164f7e P 7f20b7a1aaba41a6b2c0c190047f7f98: No bootstrap required, opened a new log
I20260926 03:31:20.739728 2641 tablet_replica.cc:439] T 8f78ad536b9b4e92afd1995e30164f7e P 7ef0f75e0d754f0597ef751b629233dc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } } }
I20260926 03:31:20.740414 2659 ts_tablet_manager.cc:1397] T 8f78ad536b9b4e92afd1995e30164f7e P 7f20b7a1aaba41a6b2c0c190047f7f98: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260926 03:31:20.740506 2641 tablet_replica.cc:442] T 8f78ad536b9b4e92afd1995e30164f7e P 7ef0f75e0d754f0597ef751b629233dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:31:20.741104 2657 ts_tablet_manager.cc:1428] T 8f78ad536b9b4e92afd1995e30164f7e P 9ca20d03820c40e6a7fe2efadf6ef182: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 03:31:20.740782 2663 tablet_replica.cc:439] T 8f78ad536b9b4e92afd1995e30164f7e P 9ca20d03820c40e6a7fe2efadf6ef182: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } } }
I20260926 03:31:20.741453 2663 tablet_replica.cc:442] T 8f78ad536b9b4e92afd1995e30164f7e P 9ca20d03820c40e6a7fe2efadf6ef182: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:31:20.741657 2659 raft_consensus.cc:348] T 8f78ad536b9b4e92afd1995e30164f7e P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:20.741890 2659 raft_consensus.cc:374] T 8f78ad536b9b4e92afd1995e30164f7e P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:20.741989 2659 raft_consensus.cc:729] T 8f78ad536b9b4e92afd1995e30164f7e P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f20b7a1aaba41a6b2c0c190047f7f98, State: Initialized, Role: FOLLOWER
I20260926 03:31:20.742278 2659 consensus_queue.cc:260] T 8f78ad536b9b4e92afd1995e30164f7e P 7f20b7a1aaba41a6b2c0c190047f7f98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:20.744328 2659 ts_tablet_manager.cc:1428] T 8f78ad536b9b4e92afd1995e30164f7e P 7f20b7a1aaba41a6b2c0c190047f7f98: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260926 03:31:20.744261 2664 tablet_replica.cc:439] T 8f78ad536b9b4e92afd1995e30164f7e P 7f20b7a1aaba41a6b2c0c190047f7f98: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } } }
I20260926 03:31:20.744724 2664 tablet_replica.cc:442] T 8f78ad536b9b4e92afd1995e30164f7e P 7f20b7a1aaba41a6b2c0c190047f7f98: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:31:20.781598 2664 raft_consensus.cc:482] T 8f78ad536b9b4e92afd1995e30164f7e P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:31:20.781838 2664 raft_consensus.cc:504] T 8f78ad536b9b4e92afd1995e30164f7e P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:20.783304 2664 leader_election.cc:288] T 8f78ad536b9b4e92afd1995e30164f7e P 7f20b7a1aaba41a6b2c0c190047f7f98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ef0f75e0d754f0597ef751b629233dc (127.0.97.2:41135), 9ca20d03820c40e6a7fe2efadf6ef182 (127.0.97.3:41599)
I20260926 03:31:20.786605 2641 raft_consensus.cc:482] T 8f78ad536b9b4e92afd1995e30164f7e P 7ef0f75e0d754f0597ef751b629233dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:31:20.786779 2641 raft_consensus.cc:504] T 8f78ad536b9b4e92afd1995e30164f7e P 7ef0f75e0d754f0597ef751b629233dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:20.787472 2641 leader_election.cc:288] T 8f78ad536b9b4e92afd1995e30164f7e P 7ef0f75e0d754f0597ef751b629233dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f20b7a1aaba41a6b2c0c190047f7f98 (127.0.97.1:46239), 9ca20d03820c40e6a7fe2efadf6ef182 (127.0.97.3:41599)
I20260926 03:31:20.787889 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f78ad536b9b4e92afd1995e30164f7e" candidate_uuid: "7ef0f75e0d754f0597ef751b629233dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" is_pre_election: true
I20260926 03:31:20.788077 2569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f78ad536b9b4e92afd1995e30164f7e" candidate_uuid: "7ef0f75e0d754f0597ef751b629233dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" is_pre_election: true
I20260926 03:31:20.788172 2292 raft_consensus.cc:2454] T 8f78ad536b9b4e92afd1995e30164f7e P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ef0f75e0d754f0597ef751b629233dc in term 0.
I20260926 03:31:20.788333 2569 raft_consensus.cc:2454] T 8f78ad536b9b4e92afd1995e30164f7e P 9ca20d03820c40e6a7fe2efadf6ef182 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ef0f75e0d754f0597ef751b629233dc in term 0.
I20260926 03:31:20.788654 2357 leader_election.cc:302] T 8f78ad536b9b4e92afd1995e30164f7e P 7ef0f75e0d754f0597ef751b629233dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ef0f75e0d754f0597ef751b629233dc, 7f20b7a1aaba41a6b2c0c190047f7f98; no voters:
I20260926 03:31:20.788949 2641 raft_consensus.cc:2781] T 8f78ad536b9b4e92afd1995e30164f7e P 7ef0f75e0d754f0597ef751b629233dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:31:20.789074 2641 raft_consensus.cc:482] T 8f78ad536b9b4e92afd1995e30164f7e P 7ef0f75e0d754f0597ef751b629233dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:31:20.789235 2641 raft_consensus.cc:3037] T 8f78ad536b9b4e92afd1995e30164f7e P 7ef0f75e0d754f0597ef751b629233dc [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:20.791020 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:20.783689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48731 (local address 127.0.97.2:41135)
0926 03:31:20.784202 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:20.784210 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:20.784328 (+ 118us) server_negotiation.cc:407] Connection header received
0926 03:31:20.784607 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:20.784612 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:20.784688 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:20.784795 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:31:20.785425 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.786163 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.788724 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.790009 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.790143 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:20.790284 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:20.790514 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 03:31:20.790643 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":251,"threads_started":1}
I20260926 03:31:20.791024 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:20.783531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41135 (local address 127.0.97.1:48731)
0926 03:31:20.784202 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:20.784222 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:20.784403 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:20.784816 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:20.784821 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:20.784849 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:31:20.785247 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:20.785257 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.786285 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.786292 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:20.788496 (+ 2204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:20.788510 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.790186 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.790192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:20.790367 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:20.790390 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 03:31:20.790492 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 03:31:20.790610 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"mutex_wait_us":50,"thread_start_us":308,"threads_started":1}
I20260926 03:31:20.791802 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f78ad536b9b4e92afd1995e30164f7e" candidate_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef0f75e0d754f0597ef751b629233dc" is_pre_election: true
I20260926 03:31:20.792263 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:20.784307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41599 (local address 127.0.97.1:44307)
0926 03:31:20.785216 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:20.785229 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:20.785390 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:20.785692 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:20.785696 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:20.785719 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:31:20.786022 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:20.786030 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.787243 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.787249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:20.789682 (+ 2433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:20.789698 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.791427 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.791443 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:20.791621 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:20.791651 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 03:31:20.791774 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 03:31:20.791887 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"mutex_wait_us":61,"thread_start_us":571,"threads_started":1}
I20260926 03:31:20.792273 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:20.784492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44307 (local address 127.0.97.3:41599)
0926 03:31:20.784743 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:20.784752 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:20.785329 (+ 577us) server_negotiation.cc:407] Connection header received
0926 03:31:20.785513 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:20.785517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:20.785587 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:20.785680 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:31:20.786159 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.787093 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.789841 (+ 2748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.791023 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.791162 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:20.791257 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:20.791799 (+ 542us) server_negotiation.cc:299] Negotiation successful
0926 03:31:20.791922 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 03:31:20.792671 2641 raft_consensus.cc:504] T 8f78ad536b9b4e92afd1995e30164f7e P 7ef0f75e0d754f0597ef751b629233dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:20.793036 2569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f78ad536b9b4e92afd1995e30164f7e" candidate_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" is_pre_election: true
I20260926 03:31:20.793257 2569 raft_consensus.cc:2454] T 8f78ad536b9b4e92afd1995e30164f7e P 9ca20d03820c40e6a7fe2efadf6ef182 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f20b7a1aaba41a6b2c0c190047f7f98 in term 0.
I20260926 03:31:20.793718 2426 raft_consensus.cc:2379] T 8f78ad536b9b4e92afd1995e30164f7e P 7ef0f75e0d754f0597ef751b629233dc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7f20b7a1aaba41a6b2c0c190047f7f98 in current term 1: Already voted for candidate 7ef0f75e0d754f0597ef751b629233dc in this term.
I20260926 03:31:20.793717 2218 leader_election.cc:302] T 8f78ad536b9b4e92afd1995e30164f7e P 7f20b7a1aaba41a6b2c0c190047f7f98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f20b7a1aaba41a6b2c0c190047f7f98, 9ca20d03820c40e6a7fe2efadf6ef182; no voters:
I20260926 03:31:20.794135 2664 raft_consensus.cc:2781] T 8f78ad536b9b4e92afd1995e30164f7e P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:31:20.794276 2664 raft_consensus.cc:482] T 8f78ad536b9b4e92afd1995e30164f7e P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:31:20.794360 2641 leader_election.cc:288] T 8f78ad536b9b4e92afd1995e30164f7e P 7ef0f75e0d754f0597ef751b629233dc [CANDIDATE]: Term 1 election: Requested vote from peers 7f20b7a1aaba41a6b2c0c190047f7f98 (127.0.97.1:46239), 9ca20d03820c40e6a7fe2efadf6ef182 (127.0.97.3:41599)
I20260926 03:31:20.794375 2664 raft_consensus.cc:3037] T 8f78ad536b9b4e92afd1995e30164f7e P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:20.794428 2569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f78ad536b9b4e92afd1995e30164f7e" candidate_uuid: "7ef0f75e0d754f0597ef751b629233dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca20d03820c40e6a7fe2efadf6ef182"
I20260926 03:31:20.794620 2569 raft_consensus.cc:3037] T 8f78ad536b9b4e92afd1995e30164f7e P 9ca20d03820c40e6a7fe2efadf6ef182 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:20.794777 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f78ad536b9b4e92afd1995e30164f7e" candidate_uuid: "7ef0f75e0d754f0597ef751b629233dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98"
I20260926 03:31:20.797374 2569 raft_consensus.cc:2454] T 8f78ad536b9b4e92afd1995e30164f7e P 9ca20d03820c40e6a7fe2efadf6ef182 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ef0f75e0d754f0597ef751b629233dc in term 1.
I20260926 03:31:20.797410 2664 raft_consensus.cc:504] T 8f78ad536b9b4e92afd1995e30164f7e P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:20.797765 2358 leader_election.cc:302] T 8f78ad536b9b4e92afd1995e30164f7e P 7ef0f75e0d754f0597ef751b629233dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ef0f75e0d754f0597ef751b629233dc, 9ca20d03820c40e6a7fe2efadf6ef182; no voters:
I20260926 03:31:20.798038 2664 leader_election.cc:288] T 8f78ad536b9b4e92afd1995e30164f7e P 7f20b7a1aaba41a6b2c0c190047f7f98 [CANDIDATE]: Term 1 election: Requested vote from peers 7ef0f75e0d754f0597ef751b629233dc (127.0.97.2:41135), 9ca20d03820c40e6a7fe2efadf6ef182 (127.0.97.3:41599)
I20260926 03:31:20.798128 2292 raft_consensus.cc:2379] T 8f78ad536b9b4e92afd1995e30164f7e P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7ef0f75e0d754f0597ef751b629233dc in current term 1: Already voted for candidate 7f20b7a1aaba41a6b2c0c190047f7f98 in this term.
I20260926 03:31:20.798411 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f78ad536b9b4e92afd1995e30164f7e" candidate_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef0f75e0d754f0597ef751b629233dc"
I20260926 03:31:20.798467 2641 raft_consensus.cc:2781] T 8f78ad536b9b4e92afd1995e30164f7e P 7ef0f75e0d754f0597ef751b629233dc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:31:20.798506 2569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f78ad536b9b4e92afd1995e30164f7e" candidate_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca20d03820c40e6a7fe2efadf6ef182"
I20260926 03:31:20.798851 2569 raft_consensus.cc:2379] T 8f78ad536b9b4e92afd1995e30164f7e P 9ca20d03820c40e6a7fe2efadf6ef182 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f20b7a1aaba41a6b2c0c190047f7f98 in current term 1: Already voted for candidate 7ef0f75e0d754f0597ef751b629233dc in this term.
I20260926 03:31:20.799310 2641 raft_consensus.cc:686] T 8f78ad536b9b4e92afd1995e30164f7e P 7ef0f75e0d754f0597ef751b629233dc [term 1 LEADER]: Becoming Leader. State: Replica: 7ef0f75e0d754f0597ef751b629233dc, State: Running, Role: LEADER
I20260926 03:31:20.799587 2641 consensus_queue.cc:237] T 8f78ad536b9b4e92afd1995e30164f7e P 7ef0f75e0d754f0597ef751b629233dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:20.801231 2645 tablet_replica.cc:439] T 8f78ad536b9b4e92afd1995e30164f7e P 7ef0f75e0d754f0597ef751b629233dc: TxnStatusTablet state changed. Reason: New leader 7ef0f75e0d754f0597ef751b629233dc. Latest consensus state: current_term: 1 leader_uuid: "7ef0f75e0d754f0597ef751b629233dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } } }
I20260926 03:31:20.801594 2645 tablet_replica.cc:442] T 8f78ad536b9b4e92afd1995e30164f7e P 7ef0f75e0d754f0597ef751b629233dc: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:31:20.802158 2218 leader_election.cc:302] T 8f78ad536b9b4e92afd1995e30164f7e P 7f20b7a1aaba41a6b2c0c190047f7f98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f20b7a1aaba41a6b2c0c190047f7f98; no voters: 7ef0f75e0d754f0597ef751b629233dc, 9ca20d03820c40e6a7fe2efadf6ef182
I20260926 03:31:20.802265 2669 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 03:31:20.802536 2669 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 03:31:20.802551 2664 raft_consensus.cc:2726] T 8f78ad536b9b4e92afd1995e30164f7e P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260926 03:31:20.802901 2128 catalog_manager.cc:5495] T 8f78ad536b9b4e92afd1995e30164f7e P 7ef0f75e0d754f0597ef751b629233dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ef0f75e0d754f0597ef751b629233dc (127.0.97.2). New cstate: current_term: 1 leader_uuid: "7ef0f75e0d754f0597ef751b629233dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } health_report { overall_health: UNKNOWN } } }
I20260926 03:31:20.821485 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:20.816028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41135 (local address 127.0.0.1:58680)
0926 03:31:20.816268 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:20.816288 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:20.816448 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:20.816923 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:20.816927 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:20.816938 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:31:20.817130 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:20.817138 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.818136 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.818143 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:20.819123 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:20.819133 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.819834 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.819839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:20.820016 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:20.820064 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:31:20.820959 (+ 895us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:31:20.820978 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 03:31:20.821048 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 03:31:20.821186 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 03:31:20.821486 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:20.816152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58680 (local address 127.0.97.2:41135)
0926 03:31:20.816351 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:20.816357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:20.816378 (+ 21us) server_negotiation.cc:407] Connection header received
0926 03:31:20.816681 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:20.816684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:20.816752 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:20.816894 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:31:20.817254 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.818013 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.819260 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.819653 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.819807 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:20.820249 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:31:20.820849 (+ 600us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:31:20.821010 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:20.821073 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 03:31:20.821146 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 03:31:20.828825 2292 raft_consensus.cc:1261] T 8f78ad536b9b4e92afd1995e30164f7e P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 1 FOLLOWER]: Refusing update from remote peer 7ef0f75e0d754f0597ef751b629233dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:31:20.828840 2569 raft_consensus.cc:1261] T 8f78ad536b9b4e92afd1995e30164f7e P 9ca20d03820c40e6a7fe2efadf6ef182 [term 1 FOLLOWER]: Refusing update from remote peer 7ef0f75e0d754f0597ef751b629233dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:31:20.829589 2645 consensus_queue.cc:1035] T 8f78ad536b9b4e92afd1995e30164f7e P 7ef0f75e0d754f0597ef751b629233dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:31:20.829943 2641 consensus_queue.cc:1035] T 8f78ad536b9b4e92afd1995e30164f7e P 7ef0f75e0d754f0597ef751b629233dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:31:20.836771 2664 tablet_replica.cc:439] T 8f78ad536b9b4e92afd1995e30164f7e P 7f20b7a1aaba41a6b2c0c190047f7f98: TxnStatusTablet state changed. Reason: New leader 7ef0f75e0d754f0597ef751b629233dc. Latest consensus state: current_term: 1 leader_uuid: "7ef0f75e0d754f0597ef751b629233dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } } }
I20260926 03:31:20.837105 2664 tablet_replica.cc:442] T 8f78ad536b9b4e92afd1995e30164f7e P 7f20b7a1aaba41a6b2c0c190047f7f98: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:31:20.840224 2645 tablet_replica.cc:439] T 8f78ad536b9b4e92afd1995e30164f7e P 7ef0f75e0d754f0597ef751b629233dc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7ef0f75e0d754f0597ef751b629233dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } } }
I20260926 03:31:20.840466 2645 tablet_replica.cc:442] T 8f78ad536b9b4e92afd1995e30164f7e P 7ef0f75e0d754f0597ef751b629233dc: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:31:20.841598 2672 mvcc.cc:205] Tried to move back new op lower bound from 7333451697465835520 to 7333451697356967936. Current Snapshot: MvccSnapshot[applied={T|T < 7333451697465835520}]
I20260926 03:31:20.850732 2664 tablet_replica.cc:439] T 8f78ad536b9b4e92afd1995e30164f7e P 7f20b7a1aaba41a6b2c0c190047f7f98: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7ef0f75e0d754f0597ef751b629233dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } } }
I20260926 03:31:20.851069 2664 tablet_replica.cc:442] T 8f78ad536b9b4e92afd1995e30164f7e P 7f20b7a1aaba41a6b2c0c190047f7f98: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:31:20.854772 2663 tablet_replica.cc:439] T 8f78ad536b9b4e92afd1995e30164f7e P 9ca20d03820c40e6a7fe2efadf6ef182: TxnStatusTablet state changed. Reason: New leader 7ef0f75e0d754f0597ef751b629233dc. Latest consensus state: current_term: 1 leader_uuid: "7ef0f75e0d754f0597ef751b629233dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } } }
I20260926 03:31:20.855077 2663 tablet_replica.cc:442] T 8f78ad536b9b4e92afd1995e30164f7e P 9ca20d03820c40e6a7fe2efadf6ef182: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:31:20.856765 2641 tablet_replica.cc:439] T 8f78ad536b9b4e92afd1995e30164f7e P 7ef0f75e0d754f0597ef751b629233dc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7ef0f75e0d754f0597ef751b629233dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } } }
I20260926 03:31:20.857095 2641 tablet_replica.cc:442] T 8f78ad536b9b4e92afd1995e30164f7e P 7ef0f75e0d754f0597ef751b629233dc: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:31:20.859766 2674 mvcc.cc:205] Tried to move back new op lower bound from 7333451697465835520 to 7333451697356967936. Current Snapshot: MvccSnapshot[applied={T|T < 7333451697465835520}]
I20260926 03:31:20.861655 2663 tablet_replica.cc:439] T 8f78ad536b9b4e92afd1995e30164f7e P 9ca20d03820c40e6a7fe2efadf6ef182: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7ef0f75e0d754f0597ef751b629233dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } } }
I20260926 03:31:20.861907 2663 tablet_replica.cc:442] T 8f78ad536b9b4e92afd1995e30164f7e P 9ca20d03820c40e6a7fe2efadf6ef182: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:31:20.862351 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:20.853717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58690 (local address 127.0.97.2:41135)
0926 03:31:20.853927 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:20.853933 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:20.856285 (+ 2352us) server_negotiation.cc:407] Connection header received
0926 03:31:20.856608 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:20.856613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:20.856724 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:20.856799 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:31:20.857474 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.858484 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.859897 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.860791 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.860903 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:20.861470 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:31:20.861644 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:31:20.861710 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:20.861947 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 03:31:20.862025 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 03:31:20.879581 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:20.872812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45307 (local address 127.0.97.2:41135)
0926 03:31:20.872983 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:20.872989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:20.873470 (+ 481us) server_negotiation.cc:407] Connection header received
0926 03:31:20.873637 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:20.873643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:20.873719 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:20.873845 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:31:20.874438 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.875602 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.877334 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.878016 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.878144 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:20.878782 (+ 638us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:31:20.878911 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:31:20.879010 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:20.879183 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 03:31:20.879256 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":47}
I20260926 03:31:20.879607 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:20.872763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41135 (local address 127.0.97.2:45307)
0926 03:31:20.873372 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:20.873386 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:20.873501 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:20.873848 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:20.873851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:20.873859 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:31:20.874271 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:20.874279 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.875769 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.875773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:20.877216 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:20.877228 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:20.878208 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:20.878218 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:20.878526 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:20.878583 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:31:20.879050 (+ 467us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:31:20.879061 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 03:31:20.879138 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 03:31:20.879243 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":17,"thread_start_us":316,"threads_started":1}
I20260926 03:31:20.889906 2568 raft_consensus.cc:1261] T ce157a73fb324adfa46a1864fd75297c P 9ca20d03820c40e6a7fe2efadf6ef182 [term 1 FOLLOWER]: Refusing update from remote peer 7ef0f75e0d754f0597ef751b629233dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:31:20.889936 2292 raft_consensus.cc:1261] T ce157a73fb324adfa46a1864fd75297c P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 1 FOLLOWER]: Refusing update from remote peer 7ef0f75e0d754f0597ef751b629233dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:31:20.890480 2645 consensus_queue.cc:1035] T ce157a73fb324adfa46a1864fd75297c P 7ef0f75e0d754f0597ef751b629233dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:31:20.890763 2641 consensus_queue.cc:1035] T ce157a73fb324adfa46a1864fd75297c P 7ef0f75e0d754f0597ef751b629233dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:31:20.903201 2673 mvcc.cc:205] Tried to move back new op lower bound from 7333451697719189504 to 7333451696509337600. Current Snapshot: MvccSnapshot[applied={T|T < 7333451697719189504}]
I20260926 03:31:20.950503 2128 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:31:20.950781 2128 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:31:20.954272 2128 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 726f37f70bde44e699103131c029838d: Sending DeleteTablet for 3 replicas of tablet ce157a73fb324adfa46a1864fd75297c
I20260926 03:31:20.955464 2272 tablet_service.cc:1514] Processing DeleteTablet for tablet ce157a73fb324adfa46a1864fd75297c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:31:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55762
I20260926 03:31:20.955631 2405 tablet_service.cc:1514] Processing DeleteTablet for tablet ce157a73fb324adfa46a1864fd75297c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:31:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58674
I20260926 03:31:20.956357 2545 tablet_service.cc:1514] Processing DeleteTablet for tablet ce157a73fb324adfa46a1864fd75297c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:31:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39320
I20260926 03:31:20.956401 2128 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:31:20.957397 2128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:31:20.960237 2690 tablet_replica.cc:330] T ce157a73fb324adfa46a1864fd75297c P 7ef0f75e0d754f0597ef751b629233dc: stopping tablet replica
I20260926 03:31:20.960743 2690 raft_consensus.cc:2229] T ce157a73fb324adfa46a1864fd75297c P 7ef0f75e0d754f0597ef751b629233dc [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:31:20.961477 2690 raft_consensus.cc:2258] T ce157a73fb324adfa46a1864fd75297c P 7ef0f75e0d754f0597ef751b629233dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:31:20.968039 2271 tablet_service.cc:1467] Processing CreateTablet for tablet d4aff8cb171348e29eea380dd84e96a3 (DEFAULT_TABLE table=test-table [id=33f5c2c5b5974894b6f0a90ff4ddcac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:31:20.968230 2689 tablet_replica.cc:330] T ce157a73fb324adfa46a1864fd75297c P 7f20b7a1aaba41a6b2c0c190047f7f98: stopping tablet replica
I20260926 03:31:20.968782 2271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4aff8cb171348e29eea380dd84e96a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:20.968824 2689 raft_consensus.cc:2229] T ce157a73fb324adfa46a1864fd75297c P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:31:20.969219 2689 raft_consensus.cc:2258] T ce157a73fb324adfa46a1864fd75297c P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:31:20.971213 2689 ts_tablet_manager.cc:1905] T ce157a73fb324adfa46a1864fd75297c P 7f20b7a1aaba41a6b2c0c190047f7f98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:31:20.973155 2691 tablet_replica.cc:330] T ce157a73fb324adfa46a1864fd75297c P 9ca20d03820c40e6a7fe2efadf6ef182: stopping tablet replica
I20260926 03:31:20.973312 2544 tablet_service.cc:1467] Processing CreateTablet for tablet d4aff8cb171348e29eea380dd84e96a3 (DEFAULT_TABLE table=test-table [id=33f5c2c5b5974894b6f0a90ff4ddcac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:31:20.973603 2691 raft_consensus.cc:2229] T ce157a73fb324adfa46a1864fd75297c P 9ca20d03820c40e6a7fe2efadf6ef182 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:31:20.973870 2544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4aff8cb171348e29eea380dd84e96a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:20.974124 2405 tablet_service.cc:1467] Processing CreateTablet for tablet d4aff8cb171348e29eea380dd84e96a3 (DEFAULT_TABLE table=test-table [id=33f5c2c5b5974894b6f0a90ff4ddcac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:31:20.974608 2405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4aff8cb171348e29eea380dd84e96a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:20.980149 2657 tablet_bootstrap.cc:492] T d4aff8cb171348e29eea380dd84e96a3 P 9ca20d03820c40e6a7fe2efadf6ef182: Bootstrap starting.
I20260926 03:31:20.980645 2659 tablet_bootstrap.cc:492] T d4aff8cb171348e29eea380dd84e96a3 P 7f20b7a1aaba41a6b2c0c190047f7f98: Bootstrap starting.
I20260926 03:31:20.981796 2657 tablet_bootstrap.cc:654] T d4aff8cb171348e29eea380dd84e96a3 P 9ca20d03820c40e6a7fe2efadf6ef182: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:20.982256 2659 tablet_bootstrap.cc:654] T d4aff8cb171348e29eea380dd84e96a3 P 7f20b7a1aaba41a6b2c0c190047f7f98: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:20.983347 2690 ts_tablet_manager.cc:1905] T ce157a73fb324adfa46a1864fd75297c P 7ef0f75e0d754f0597ef751b629233dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:31:20.985285 2658 tablet_bootstrap.cc:492] T d4aff8cb171348e29eea380dd84e96a3 P 7ef0f75e0d754f0597ef751b629233dc: Bootstrap starting.
I20260926 03:31:20.987483 2658 tablet_bootstrap.cc:654] T d4aff8cb171348e29eea380dd84e96a3 P 7ef0f75e0d754f0597ef751b629233dc: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:20.992717 2691 raft_consensus.cc:2258] T ce157a73fb324adfa46a1864fd75297c P 9ca20d03820c40e6a7fe2efadf6ef182 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:31:20.995887 2691 ts_tablet_manager.cc:1905] T ce157a73fb324adfa46a1864fd75297c P 9ca20d03820c40e6a7fe2efadf6ef182: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:31:21.005241 2658 tablet_bootstrap.cc:492] T d4aff8cb171348e29eea380dd84e96a3 P 7ef0f75e0d754f0597ef751b629233dc: No bootstrap required, opened a new log
I20260926 03:31:21.005573 2658 ts_tablet_manager.cc:1397] T d4aff8cb171348e29eea380dd84e96a3 P 7ef0f75e0d754f0597ef751b629233dc: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.009s
I20260926 03:31:21.006733 2658 raft_consensus.cc:348] T d4aff8cb171348e29eea380dd84e96a3 P 7ef0f75e0d754f0597ef751b629233dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:21.007221 2658 raft_consensus.cc:374] T d4aff8cb171348e29eea380dd84e96a3 P 7ef0f75e0d754f0597ef751b629233dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:21.008415 2658 raft_consensus.cc:729] T d4aff8cb171348e29eea380dd84e96a3 P 7ef0f75e0d754f0597ef751b629233dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef0f75e0d754f0597ef751b629233dc, State: Initialized, Role: FOLLOWER
I20260926 03:31:21.008849 2658 consensus_queue.cc:260] T d4aff8cb171348e29eea380dd84e96a3 P 7ef0f75e0d754f0597ef751b629233dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:21.009490 2658 ts_tablet_manager.cc:1428] T d4aff8cb171348e29eea380dd84e96a3 P 7ef0f75e0d754f0597ef751b629233dc: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260926 03:31:21.012166 2690 ts_tablet_manager.cc:1918] T ce157a73fb324adfa46a1864fd75297c P 7ef0f75e0d754f0597ef751b629233dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:31:21.012415 2690 log.cc:1196] T ce157a73fb324adfa46a1864fd75297c P 7ef0f75e0d754f0597ef751b629233dc: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-1/wal/wals/ce157a73fb324adfa46a1864fd75297c
I20260926 03:31:21.013067 2690 ts_tablet_manager.cc:1939] T ce157a73fb324adfa46a1864fd75297c P 7ef0f75e0d754f0597ef751b629233dc: Deleting consensus metadata
I20260926 03:31:21.014048 2691 ts_tablet_manager.cc:1918] T ce157a73fb324adfa46a1864fd75297c P 9ca20d03820c40e6a7fe2efadf6ef182: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:31:21.014533 2691 log.cc:1196] T ce157a73fb324adfa46a1864fd75297c P 9ca20d03820c40e6a7fe2efadf6ef182: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-2/wal/wals/ce157a73fb324adfa46a1864fd75297c
I20260926 03:31:21.015090 2691 ts_tablet_manager.cc:1939] T ce157a73fb324adfa46a1864fd75297c P 9ca20d03820c40e6a7fe2efadf6ef182: Deleting consensus metadata
I20260926 03:31:21.015321 2115 catalog_manager.cc:4841] TS 7ef0f75e0d754f0597ef751b629233dc (127.0.97.2:41135): tablet ce157a73fb324adfa46a1864fd75297c (table test-table [id=d325ef6e81ad498190f0f5f7151a592c]) successfully deleted
I20260926 03:31:21.017937 2115 catalog_manager.cc:4841] TS 9ca20d03820c40e6a7fe2efadf6ef182 (127.0.97.3:41599): tablet ce157a73fb324adfa46a1864fd75297c (table test-table [id=d325ef6e81ad498190f0f5f7151a592c]) successfully deleted
I20260926 03:31:21.025549 2659 tablet_bootstrap.cc:492] T d4aff8cb171348e29eea380dd84e96a3 P 7f20b7a1aaba41a6b2c0c190047f7f98: No bootstrap required, opened a new log
I20260926 03:31:21.025791 2659 ts_tablet_manager.cc:1397] T d4aff8cb171348e29eea380dd84e96a3 P 7f20b7a1aaba41a6b2c0c190047f7f98: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.025s
I20260926 03:31:21.027726 2659 raft_consensus.cc:348] T d4aff8cb171348e29eea380dd84e96a3 P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:21.028060 2659 raft_consensus.cc:374] T d4aff8cb171348e29eea380dd84e96a3 P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:21.028617 2659 raft_consensus.cc:729] T d4aff8cb171348e29eea380dd84e96a3 P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f20b7a1aaba41a6b2c0c190047f7f98, State: Initialized, Role: FOLLOWER
I20260926 03:31:21.028959 2659 consensus_queue.cc:260] T d4aff8cb171348e29eea380dd84e96a3 P 7f20b7a1aaba41a6b2c0c190047f7f98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:21.031482 2659 ts_tablet_manager.cc:1428] T d4aff8cb171348e29eea380dd84e96a3 P 7f20b7a1aaba41a6b2c0c190047f7f98: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260926 03:31:21.031059 2689 ts_tablet_manager.cc:1918] T ce157a73fb324adfa46a1864fd75297c P 7f20b7a1aaba41a6b2c0c190047f7f98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:31:21.031798 2689 log.cc:1196] T ce157a73fb324adfa46a1864fd75297c P 7f20b7a1aaba41a6b2c0c190047f7f98: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-0/wal/wals/ce157a73fb324adfa46a1864fd75297c
I20260926 03:31:21.032362 2689 ts_tablet_manager.cc:1939] T ce157a73fb324adfa46a1864fd75297c P 7f20b7a1aaba41a6b2c0c190047f7f98: Deleting consensus metadata
I20260926 03:31:21.034117 2114 catalog_manager.cc:4841] TS 7f20b7a1aaba41a6b2c0c190047f7f98 (127.0.97.1:46239): tablet ce157a73fb324adfa46a1864fd75297c (table test-table [id=d325ef6e81ad498190f0f5f7151a592c]) successfully deleted
I20260926 03:31:21.035432 2657 tablet_bootstrap.cc:492] T d4aff8cb171348e29eea380dd84e96a3 P 9ca20d03820c40e6a7fe2efadf6ef182: No bootstrap required, opened a new log
I20260926 03:31:21.035617 2657 ts_tablet_manager.cc:1397] T d4aff8cb171348e29eea380dd84e96a3 P 9ca20d03820c40e6a7fe2efadf6ef182: Time spent bootstrapping tablet: real 0.056s user 0.005s sys 0.006s
I20260926 03:31:21.036392 2657 raft_consensus.cc:348] T d4aff8cb171348e29eea380dd84e96a3 P 9ca20d03820c40e6a7fe2efadf6ef182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:21.036684 2657 raft_consensus.cc:374] T d4aff8cb171348e29eea380dd84e96a3 P 9ca20d03820c40e6a7fe2efadf6ef182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:21.036762 2657 raft_consensus.cc:729] T d4aff8cb171348e29eea380dd84e96a3 P 9ca20d03820c40e6a7fe2efadf6ef182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ca20d03820c40e6a7fe2efadf6ef182, State: Initialized, Role: FOLLOWER
I20260926 03:31:21.036988 2657 consensus_queue.cc:260] T d4aff8cb171348e29eea380dd84e96a3 P 9ca20d03820c40e6a7fe2efadf6ef182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:21.037477 2657 ts_tablet_manager.cc:1428] T d4aff8cb171348e29eea380dd84e96a3 P 9ca20d03820c40e6a7fe2efadf6ef182: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 03:31:21.193377 2687 raft_consensus.cc:482] T d4aff8cb171348e29eea380dd84e96a3 P 7ef0f75e0d754f0597ef751b629233dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:31:21.193612 2687 raft_consensus.cc:504] T d4aff8cb171348e29eea380dd84e96a3 P 7ef0f75e0d754f0597ef751b629233dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:21.194501 2687 leader_election.cc:288] T d4aff8cb171348e29eea380dd84e96a3 P 7ef0f75e0d754f0597ef751b629233dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f20b7a1aaba41a6b2c0c190047f7f98 (127.0.97.1:46239), 9ca20d03820c40e6a7fe2efadf6ef182 (127.0.97.3:41599)
I20260926 03:31:21.194977 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4aff8cb171348e29eea380dd84e96a3" candidate_uuid: "7ef0f75e0d754f0597ef751b629233dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" is_pre_election: true
I20260926 03:31:21.195061 2568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4aff8cb171348e29eea380dd84e96a3" candidate_uuid: "7ef0f75e0d754f0597ef751b629233dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" is_pre_election: true
I20260926 03:31:21.195220 2291 raft_consensus.cc:2454] T d4aff8cb171348e29eea380dd84e96a3 P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ef0f75e0d754f0597ef751b629233dc in term 0.
I20260926 03:31:21.195364 2568 raft_consensus.cc:2454] T d4aff8cb171348e29eea380dd84e96a3 P 9ca20d03820c40e6a7fe2efadf6ef182 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ef0f75e0d754f0597ef751b629233dc in term 0.
I20260926 03:31:21.195675 2357 leader_election.cc:302] T d4aff8cb171348e29eea380dd84e96a3 P 7ef0f75e0d754f0597ef751b629233dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ef0f75e0d754f0597ef751b629233dc, 7f20b7a1aaba41a6b2c0c190047f7f98; no voters:
I20260926 03:31:21.195950 2687 raft_consensus.cc:2781] T d4aff8cb171348e29eea380dd84e96a3 P 7ef0f75e0d754f0597ef751b629233dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:31:21.196081 2687 raft_consensus.cc:482] T d4aff8cb171348e29eea380dd84e96a3 P 7ef0f75e0d754f0597ef751b629233dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:31:21.196197 2687 raft_consensus.cc:3037] T d4aff8cb171348e29eea380dd84e96a3 P 7ef0f75e0d754f0597ef751b629233dc [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:21.198475 2687 raft_consensus.cc:504] T d4aff8cb171348e29eea380dd84e96a3 P 7ef0f75e0d754f0597ef751b629233dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:21.199115 2687 leader_election.cc:288] T d4aff8cb171348e29eea380dd84e96a3 P 7ef0f75e0d754f0597ef751b629233dc [CANDIDATE]: Term 1 election: Requested vote from peers 7f20b7a1aaba41a6b2c0c190047f7f98 (127.0.97.1:46239), 9ca20d03820c40e6a7fe2efadf6ef182 (127.0.97.3:41599)
I20260926 03:31:21.199584 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4aff8cb171348e29eea380dd84e96a3" candidate_uuid: "7ef0f75e0d754f0597ef751b629233dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98"
I20260926 03:31:21.199662 2568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4aff8cb171348e29eea380dd84e96a3" candidate_uuid: "7ef0f75e0d754f0597ef751b629233dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca20d03820c40e6a7fe2efadf6ef182"
I20260926 03:31:21.199785 2291 raft_consensus.cc:3037] T d4aff8cb171348e29eea380dd84e96a3 P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:21.199875 2568 raft_consensus.cc:3037] T d4aff8cb171348e29eea380dd84e96a3 P 9ca20d03820c40e6a7fe2efadf6ef182 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:21.203182 2568 raft_consensus.cc:2454] T d4aff8cb171348e29eea380dd84e96a3 P 9ca20d03820c40e6a7fe2efadf6ef182 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ef0f75e0d754f0597ef751b629233dc in term 1.
I20260926 03:31:21.203224 2291 raft_consensus.cc:2454] T d4aff8cb171348e29eea380dd84e96a3 P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ef0f75e0d754f0597ef751b629233dc in term 1.
I20260926 03:31:21.203689 2358 leader_election.cc:302] T d4aff8cb171348e29eea380dd84e96a3 P 7ef0f75e0d754f0597ef751b629233dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ef0f75e0d754f0597ef751b629233dc, 9ca20d03820c40e6a7fe2efadf6ef182; no voters:
I20260926 03:31:21.204047 2687 raft_consensus.cc:2781] T d4aff8cb171348e29eea380dd84e96a3 P 7ef0f75e0d754f0597ef751b629233dc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:31:21.204275 2687 raft_consensus.cc:686] T d4aff8cb171348e29eea380dd84e96a3 P 7ef0f75e0d754f0597ef751b629233dc [term 1 LEADER]: Becoming Leader. State: Replica: 7ef0f75e0d754f0597ef751b629233dc, State: Running, Role: LEADER
I20260926 03:31:21.204947 2687 consensus_queue.cc:237] T d4aff8cb171348e29eea380dd84e96a3 P 7ef0f75e0d754f0597ef751b629233dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:21.207782 2128 catalog_manager.cc:5495] T d4aff8cb171348e29eea380dd84e96a3 P 7ef0f75e0d754f0597ef751b629233dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ef0f75e0d754f0597ef751b629233dc (127.0.97.2). New cstate: current_term: 1 leader_uuid: "7ef0f75e0d754f0597ef751b629233dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } health_report { overall_health: UNKNOWN } } }
I20260926 03:31:21.233083 2569 raft_consensus.cc:1261] T d4aff8cb171348e29eea380dd84e96a3 P 9ca20d03820c40e6a7fe2efadf6ef182 [term 1 FOLLOWER]: Refusing update from remote peer 7ef0f75e0d754f0597ef751b629233dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:31:21.233055 2291 raft_consensus.cc:1261] T d4aff8cb171348e29eea380dd84e96a3 P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 1 FOLLOWER]: Refusing update from remote peer 7ef0f75e0d754f0597ef751b629233dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:31:21.233592 2645 consensus_queue.cc:1035] T d4aff8cb171348e29eea380dd84e96a3 P 7ef0f75e0d754f0597ef751b629233dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:31:21.233848 2641 consensus_queue.cc:1035] T d4aff8cb171348e29eea380dd84e96a3 P 7ef0f75e0d754f0597ef751b629233dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 03:31:21.255896 2672 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 03:31:22.836628 2130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:31:22.836843 2130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:31:22.841073 2130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 726f37f70bde44e699103131c029838d: Sending DeleteTablet for 3 replicas of tablet d4aff8cb171348e29eea380dd84e96a3
I20260926 03:31:22.841809 2271 tablet_service.cc:1514] Processing DeleteTablet for tablet d4aff8cb171348e29eea380dd84e96a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:31:22 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55762
I20260926 03:31:22.841841 2405 tablet_service.cc:1514] Processing DeleteTablet for tablet d4aff8cb171348e29eea380dd84e96a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:31:22 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58674
I20260926 03:31:22.842010 2544 tablet_service.cc:1514] Processing DeleteTablet for tablet d4aff8cb171348e29eea380dd84e96a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:31:22 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39320
I20260926 03:31:22.854338 2720 tablet_replica.cc:330] T d4aff8cb171348e29eea380dd84e96a3 P 9ca20d03820c40e6a7fe2efadf6ef182: stopping tablet replica
I20260926 03:31:22.854337 2718 tablet_replica.cc:330] T d4aff8cb171348e29eea380dd84e96a3 P 7f20b7a1aaba41a6b2c0c190047f7f98: stopping tablet replica
I20260926 03:31:22.854605 2720 raft_consensus.cc:2229] T d4aff8cb171348e29eea380dd84e96a3 P 9ca20d03820c40e6a7fe2efadf6ef182 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:31:22.854876 2720 raft_consensus.cc:2258] T d4aff8cb171348e29eea380dd84e96a3 P 9ca20d03820c40e6a7fe2efadf6ef182 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:31:22.855115 2719 tablet_replica.cc:330] T d4aff8cb171348e29eea380dd84e96a3 P 7ef0f75e0d754f0597ef751b629233dc: stopping tablet replica
I20260926 03:31:22.855307 2719 raft_consensus.cc:2229] T d4aff8cb171348e29eea380dd84e96a3 P 7ef0f75e0d754f0597ef751b629233dc [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:31:22.855650 2719 raft_consensus.cc:2258] T d4aff8cb171348e29eea380dd84e96a3 P 7ef0f75e0d754f0597ef751b629233dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:31:22.855950 2720 ts_tablet_manager.cc:1905] T d4aff8cb171348e29eea380dd84e96a3 P 9ca20d03820c40e6a7fe2efadf6ef182: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:31:22.856032 2718 raft_consensus.cc:2229] T d4aff8cb171348e29eea380dd84e96a3 P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:31:22.856271 2718 raft_consensus.cc:2258] T d4aff8cb171348e29eea380dd84e96a3 P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:31:22.856998 2719 ts_tablet_manager.cc:1905] T d4aff8cb171348e29eea380dd84e96a3 P 7ef0f75e0d754f0597ef751b629233dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:31:22.861220 2718 ts_tablet_manager.cc:1905] T d4aff8cb171348e29eea380dd84e96a3 P 7f20b7a1aaba41a6b2c0c190047f7f98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:31:22.862453 2719 ts_tablet_manager.cc:1918] T d4aff8cb171348e29eea380dd84e96a3 P 7ef0f75e0d754f0597ef751b629233dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 03:31:22.862596 2719 log.cc:1196] T d4aff8cb171348e29eea380dd84e96a3 P 7ef0f75e0d754f0597ef751b629233dc: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-1/wal/wals/d4aff8cb171348e29eea380dd84e96a3
I20260926 03:31:22.863085 2719 ts_tablet_manager.cc:1939] T d4aff8cb171348e29eea380dd84e96a3 P 7ef0f75e0d754f0597ef751b629233dc: Deleting consensus metadata
I20260926 03:31:22.863672 2720 ts_tablet_manager.cc:1918] T d4aff8cb171348e29eea380dd84e96a3 P 9ca20d03820c40e6a7fe2efadf6ef182: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 03:31:22.863816 2720 log.cc:1196] T d4aff8cb171348e29eea380dd84e96a3 P 9ca20d03820c40e6a7fe2efadf6ef182: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-2/wal/wals/d4aff8cb171348e29eea380dd84e96a3
I20260926 03:31:22.864239 2720 ts_tablet_manager.cc:1939] T d4aff8cb171348e29eea380dd84e96a3 P 9ca20d03820c40e6a7fe2efadf6ef182: Deleting consensus metadata
I20260926 03:31:22.864212 2115 catalog_manager.cc:4841] TS 7ef0f75e0d754f0597ef751b629233dc (127.0.97.2:41135): tablet d4aff8cb171348e29eea380dd84e96a3 (table test-table [id=33f5c2c5b5974894b6f0a90ff4ddcac1]) successfully deleted
I20260926 03:31:22.865377 2115 catalog_manager.cc:4841] TS 9ca20d03820c40e6a7fe2efadf6ef182 (127.0.97.3:41599): tablet d4aff8cb171348e29eea380dd84e96a3 (table test-table [id=33f5c2c5b5974894b6f0a90ff4ddcac1]) successfully deleted
I20260926 03:31:22.868703 2718 ts_tablet_manager.cc:1918] T d4aff8cb171348e29eea380dd84e96a3 P 7f20b7a1aaba41a6b2c0c190047f7f98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 03:31:22.868840 2718 log.cc:1196] T d4aff8cb171348e29eea380dd84e96a3 P 7f20b7a1aaba41a6b2c0c190047f7f98: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-0/wal/wals/d4aff8cb171348e29eea380dd84e96a3
I20260926 03:31:22.869273 2718 ts_tablet_manager.cc:1939] T d4aff8cb171348e29eea380dd84e96a3 P 7f20b7a1aaba41a6b2c0c190047f7f98: Deleting consensus metadata
I20260926 03:31:22.870165 2114 catalog_manager.cc:4841] TS 7f20b7a1aaba41a6b2c0c190047f7f98 (127.0.97.1:46239): tablet d4aff8cb171348e29eea380dd84e96a3 (table test-table [id=33f5c2c5b5974894b6f0a90ff4ddcac1]) successfully deleted
Sep 26 03:31:22 dist-test-slave-dt7p krb5kdc[2085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393482, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 03:31:22 dist-test-slave-dt7p krb5kdc[2085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393482, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:31:22.899093 2728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:22.886071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34982 (local address 127.0.97.62:33565)
0926 03:31:22.886558 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:22.886563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:22.886628 (+ 65us) server_negotiation.cc:407] Connection header received
0926 03:31:22.886774 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:22.886778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:22.886838 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:22.886932 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:22.888282 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:22.889127 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:22.890368 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:22.890737 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:22.890829 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:22.894483 (+ 3654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:22.894488 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:22.894492 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:22.894522 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:22.897002 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:22.897570 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:22.897574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:22.897576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:22.897688 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:22.898006 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:22.898010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:22.898011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:22.898253 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:22.898415 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:22.898655 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 03:31:22.898811 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":264,"threads_started":1}
I20260926 03:31:22.902443 2130 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34982:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:31:22.903393 2130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:31:22.910187 2271 tablet_service.cc:1467] Processing CreateTablet for tablet 625893e548e14bd5b2dbe1589c7413ef (DEFAULT_TABLE table=test-table [id=71e10a6254884bc78c3ae418764084b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:31:22.910341 2405 tablet_service.cc:1467] Processing CreateTablet for tablet 625893e548e14bd5b2dbe1589c7413ef (DEFAULT_TABLE table=test-table [id=71e10a6254884bc78c3ae418764084b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:31:22.910672 2271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 625893e548e14bd5b2dbe1589c7413ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:22.910712 2405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 625893e548e14bd5b2dbe1589c7413ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:22.910732 2544 tablet_service.cc:1467] Processing CreateTablet for tablet 625893e548e14bd5b2dbe1589c7413ef (DEFAULT_TABLE table=test-table [id=71e10a6254884bc78c3ae418764084b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:31:22.911094 2544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 625893e548e14bd5b2dbe1589c7413ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:22.917709 2729 tablet_bootstrap.cc:492] T 625893e548e14bd5b2dbe1589c7413ef P 7f20b7a1aaba41a6b2c0c190047f7f98: Bootstrap starting.
I20260926 03:31:22.918349 2731 tablet_bootstrap.cc:492] T 625893e548e14bd5b2dbe1589c7413ef P 7ef0f75e0d754f0597ef751b629233dc: Bootstrap starting.
I20260926 03:31:22.920301 2731 tablet_bootstrap.cc:654] T 625893e548e14bd5b2dbe1589c7413ef P 7ef0f75e0d754f0597ef751b629233dc: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:22.920312 2729 tablet_bootstrap.cc:654] T 625893e548e14bd5b2dbe1589c7413ef P 7f20b7a1aaba41a6b2c0c190047f7f98: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:22.921092 2730 tablet_bootstrap.cc:492] T 625893e548e14bd5b2dbe1589c7413ef P 9ca20d03820c40e6a7fe2efadf6ef182: Bootstrap starting.
I20260926 03:31:22.922736 2731 tablet_bootstrap.cc:492] T 625893e548e14bd5b2dbe1589c7413ef P 7ef0f75e0d754f0597ef751b629233dc: No bootstrap required, opened a new log
I20260926 03:31:22.922739 2729 tablet_bootstrap.cc:492] T 625893e548e14bd5b2dbe1589c7413ef P 7f20b7a1aaba41a6b2c0c190047f7f98: No bootstrap required, opened a new log
I20260926 03:31:22.922798 2730 tablet_bootstrap.cc:654] T 625893e548e14bd5b2dbe1589c7413ef P 9ca20d03820c40e6a7fe2efadf6ef182: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:22.922910 2729 ts_tablet_manager.cc:1397] T 625893e548e14bd5b2dbe1589c7413ef P 7f20b7a1aaba41a6b2c0c190047f7f98: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260926 03:31:22.922923 2731 ts_tablet_manager.cc:1397] T 625893e548e14bd5b2dbe1589c7413ef P 7ef0f75e0d754f0597ef751b629233dc: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260926 03:31:22.923650 2731 raft_consensus.cc:348] T 625893e548e14bd5b2dbe1589c7413ef P 7ef0f75e0d754f0597ef751b629233dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:22.923704 2729 raft_consensus.cc:348] T 625893e548e14bd5b2dbe1589c7413ef P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:22.923949 2729 raft_consensus.cc:374] T 625893e548e14bd5b2dbe1589c7413ef P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:22.923962 2731 raft_consensus.cc:374] T 625893e548e14bd5b2dbe1589c7413ef P 7ef0f75e0d754f0597ef751b629233dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:22.924044 2729 raft_consensus.cc:729] T 625893e548e14bd5b2dbe1589c7413ef P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f20b7a1aaba41a6b2c0c190047f7f98, State: Initialized, Role: FOLLOWER
I20260926 03:31:22.924064 2731 raft_consensus.cc:729] T 625893e548e14bd5b2dbe1589c7413ef P 7ef0f75e0d754f0597ef751b629233dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef0f75e0d754f0597ef751b629233dc, State: Initialized, Role: FOLLOWER
I20260926 03:31:22.924279 2729 consensus_queue.cc:260] T 625893e548e14bd5b2dbe1589c7413ef P 7f20b7a1aaba41a6b2c0c190047f7f98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:22.924319 2731 consensus_queue.cc:260] T 625893e548e14bd5b2dbe1589c7413ef P 7ef0f75e0d754f0597ef751b629233dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:22.925037 2731 ts_tablet_manager.cc:1428] T 625893e548e14bd5b2dbe1589c7413ef P 7ef0f75e0d754f0597ef751b629233dc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 03:31:22.926210 2730 tablet_bootstrap.cc:492] T 625893e548e14bd5b2dbe1589c7413ef P 9ca20d03820c40e6a7fe2efadf6ef182: No bootstrap required, opened a new log
I20260926 03:31:22.926383 2730 ts_tablet_manager.cc:1397] T 625893e548e14bd5b2dbe1589c7413ef P 9ca20d03820c40e6a7fe2efadf6ef182: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 03:31:22.927352 2730 raft_consensus.cc:348] T 625893e548e14bd5b2dbe1589c7413ef P 9ca20d03820c40e6a7fe2efadf6ef182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:22.927645 2730 raft_consensus.cc:374] T 625893e548e14bd5b2dbe1589c7413ef P 9ca20d03820c40e6a7fe2efadf6ef182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:22.927757 2730 raft_consensus.cc:729] T 625893e548e14bd5b2dbe1589c7413ef P 9ca20d03820c40e6a7fe2efadf6ef182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ca20d03820c40e6a7fe2efadf6ef182, State: Initialized, Role: FOLLOWER
I20260926 03:31:22.928051 2730 consensus_queue.cc:260] T 625893e548e14bd5b2dbe1589c7413ef P 9ca20d03820c40e6a7fe2efadf6ef182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:22.929564 2730 ts_tablet_manager.cc:1428] T 625893e548e14bd5b2dbe1589c7413ef P 9ca20d03820c40e6a7fe2efadf6ef182: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 03:31:22.932410 2729 ts_tablet_manager.cc:1428] T 625893e548e14bd5b2dbe1589c7413ef P 7f20b7a1aaba41a6b2c0c190047f7f98: Time spent starting tablet: real 0.009s user 0.004s sys 0.005s
I20260926 03:31:22.941327 2736 raft_consensus.cc:482] T 625893e548e14bd5b2dbe1589c7413ef P 9ca20d03820c40e6a7fe2efadf6ef182 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:31:22.941545 2736 raft_consensus.cc:504] T 625893e548e14bd5b2dbe1589c7413ef P 9ca20d03820c40e6a7fe2efadf6ef182 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:22.943189 2736 leader_election.cc:288] T 625893e548e14bd5b2dbe1589c7413ef P 9ca20d03820c40e6a7fe2efadf6ef182 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f20b7a1aaba41a6b2c0c190047f7f98 (127.0.97.1:46239), 7ef0f75e0d754f0597ef751b629233dc (127.0.97.2:41135)
I20260926 03:31:22.961278 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:22.943590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42963 (local address 127.0.97.2:41135)
0926 03:31:22.955700 (+ 12110us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:22.955704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:22.955723 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:31:22.955774 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:22.955778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:22.955831 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:22.955943 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:31:22.956488 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:22.957340 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:22.959502 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:22.960266 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:22.960404 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:22.960539 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:22.960826 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 03:31:22.960950 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11956,"thread_start_us":11784,"threads_started":1}
I20260926 03:31:22.961450 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:22.943469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58181 (local address 127.0.97.1:46239)
0926 03:31:22.954761 (+ 11292us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:22.954771 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:22.956092 (+ 1321us) server_negotiation.cc:407] Connection header received
0926 03:31:22.956304 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:22.956313 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:22.956367 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:22.956471 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:31:22.957004 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:22.957717 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:22.959734 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:22.960482 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:22.960653 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:22.960761 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:22.961016 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 03:31:22.961139 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11096,"thread_start_us":11319,"threads_started":1}
I20260926 03:31:22.961864 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:22.943423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46239 (local address 127.0.97.3:58181)
0926 03:31:22.955973 (+ 12550us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:22.955988 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:22.956165 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:22.956482 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:22.956486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:22.956517 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:31:22.956837 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:22.956845 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:22.957842 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:22.957854 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:22.959589 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:22.959600 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:22.960711 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:22.960717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:22.960862 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:22.960884 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 03:31:22.960983 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 03:31:22.961098 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12425,"mutex_wait_us":45,"thread_start_us":11619,"threads_started":1}
I20260926 03:31:22.961925 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:22.943423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41135 (local address 127.0.97.3:42963)
0926 03:31:22.955344 (+ 11921us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:22.955377 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:22.955519 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:22.956005 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:22.956010 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:22.956041 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:31:22.956303 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:22.956311 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:22.957491 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:22.957497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:22.959355 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:22.959366 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:22.960452 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:22.960458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:22.960675 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:22.960702 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 03:31:22.960803 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 03:31:22.960929 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11775,"mutex_wait_us":43,"thread_start_us":12252,"threads_started":1}
I20260926 03:31:22.962481 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625893e548e14bd5b2dbe1589c7413ef" candidate_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" is_pre_election: true
I20260926 03:31:22.962662 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625893e548e14bd5b2dbe1589c7413ef" candidate_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef0f75e0d754f0597ef751b629233dc" is_pre_election: true
I20260926 03:31:22.962738 2291 raft_consensus.cc:2454] T 625893e548e14bd5b2dbe1589c7413ef P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ca20d03820c40e6a7fe2efadf6ef182 in term 0.
I20260926 03:31:22.962915 2426 raft_consensus.cc:2454] T 625893e548e14bd5b2dbe1589c7413ef P 7ef0f75e0d754f0597ef751b629233dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ca20d03820c40e6a7fe2efadf6ef182 in term 0.
I20260926 03:31:22.963228 2496 leader_election.cc:302] T 625893e548e14bd5b2dbe1589c7413ef P 9ca20d03820c40e6a7fe2efadf6ef182 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f20b7a1aaba41a6b2c0c190047f7f98, 9ca20d03820c40e6a7fe2efadf6ef182; no voters:
I20260926 03:31:22.963622 2736 raft_consensus.cc:2781] T 625893e548e14bd5b2dbe1589c7413ef P 9ca20d03820c40e6a7fe2efadf6ef182 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:31:22.963763 2736 raft_consensus.cc:482] T 625893e548e14bd5b2dbe1589c7413ef P 9ca20d03820c40e6a7fe2efadf6ef182 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:31:22.963864 2736 raft_consensus.cc:3037] T 625893e548e14bd5b2dbe1589c7413ef P 9ca20d03820c40e6a7fe2efadf6ef182 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:22.966104 2736 raft_consensus.cc:504] T 625893e548e14bd5b2dbe1589c7413ef P 9ca20d03820c40e6a7fe2efadf6ef182 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:22.966781 2736 leader_election.cc:288] T 625893e548e14bd5b2dbe1589c7413ef P 9ca20d03820c40e6a7fe2efadf6ef182 [CANDIDATE]: Term 1 election: Requested vote from peers 7f20b7a1aaba41a6b2c0c190047f7f98 (127.0.97.1:46239), 7ef0f75e0d754f0597ef751b629233dc (127.0.97.2:41135)
I20260926 03:31:22.967134 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625893e548e14bd5b2dbe1589c7413ef" candidate_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98"
I20260926 03:31:22.967218 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625893e548e14bd5b2dbe1589c7413ef" candidate_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef0f75e0d754f0597ef751b629233dc"
I20260926 03:31:22.967340 2291 raft_consensus.cc:3037] T 625893e548e14bd5b2dbe1589c7413ef P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:22.967399 2426 raft_consensus.cc:3037] T 625893e548e14bd5b2dbe1589c7413ef P 7ef0f75e0d754f0597ef751b629233dc [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:22.970299 2291 raft_consensus.cc:2454] T 625893e548e14bd5b2dbe1589c7413ef P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ca20d03820c40e6a7fe2efadf6ef182 in term 1.
I20260926 03:31:22.970291 2426 raft_consensus.cc:2454] T 625893e548e14bd5b2dbe1589c7413ef P 7ef0f75e0d754f0597ef751b629233dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ca20d03820c40e6a7fe2efadf6ef182 in term 1.
I20260926 03:31:22.970678 2497 leader_election.cc:302] T 625893e548e14bd5b2dbe1589c7413ef P 9ca20d03820c40e6a7fe2efadf6ef182 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ef0f75e0d754f0597ef751b629233dc, 9ca20d03820c40e6a7fe2efadf6ef182; no voters:
I20260926 03:31:22.970986 2736 raft_consensus.cc:2781] T 625893e548e14bd5b2dbe1589c7413ef P 9ca20d03820c40e6a7fe2efadf6ef182 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:31:22.971347 2736 raft_consensus.cc:686] T 625893e548e14bd5b2dbe1589c7413ef P 9ca20d03820c40e6a7fe2efadf6ef182 [term 1 LEADER]: Becoming Leader. State: Replica: 9ca20d03820c40e6a7fe2efadf6ef182, State: Running, Role: LEADER
I20260926 03:31:22.972113 2736 consensus_queue.cc:237] T 625893e548e14bd5b2dbe1589c7413ef P 9ca20d03820c40e6a7fe2efadf6ef182 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:22.975373 2128 catalog_manager.cc:5495] T 625893e548e14bd5b2dbe1589c7413ef P 9ca20d03820c40e6a7fe2efadf6ef182 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ca20d03820c40e6a7fe2efadf6ef182 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } health_report { overall_health: HEALTHY } } }
I20260926 03:31:23.014870 2744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:23.004702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39108 (local address 127.0.97.3:41599)
0926 03:31:23.009002 (+ 4300us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:23.009008 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:23.009030 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:31:23.009128 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:23.009133 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:23.009234 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:23.009406 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:31:23.009767 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:23.010975 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:23.012207 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:23.012888 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:23.012994 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:23.013410 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:31:23.014201 (+ 791us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:31:23.014331 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:23.014420 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 03:31:23.014514 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4170,"thread_start_us":4036,"threads_started":1}
I20260926 03:31:23.029788 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:23.024942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35241 (local address 127.0.97.2:41135)
0926 03:31:23.025115 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:23.025119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:23.025569 (+ 450us) server_negotiation.cc:407] Connection header received
0926 03:31:23.025749 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:23.025753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:23.025817 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:23.025896 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:31:23.026338 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:23.027110 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:23.028370 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:23.028808 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:23.028896 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:23.029263 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:31:23.029357 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:31:23.029419 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:23.029543 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 03:31:23.029600 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 03:31:23.029860 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:23.024832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41135 (local address 127.0.97.3:35241)
0926 03:31:23.025465 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:23.025481 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:23.025616 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:23.025926 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:23.025929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:23.025938 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:31:23.026207 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:23.026215 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:23.027225 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:23.027229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:23.028245 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:23.028255 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:23.028956 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:23.028960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:23.029075 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:23.029123 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:31:23.029435 (+ 312us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:31:23.029442 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 03:31:23.029503 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 03:31:23.029606 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":326,"threads_started":1}
I20260926 03:31:23.037452 2291 raft_consensus.cc:1261] T 625893e548e14bd5b2dbe1589c7413ef P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 1 FOLLOWER]: Refusing update from remote peer 9ca20d03820c40e6a7fe2efadf6ef182: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:31:23.037570 2426 raft_consensus.cc:1261] T 625893e548e14bd5b2dbe1589c7413ef P 7ef0f75e0d754f0597ef751b629233dc [term 1 FOLLOWER]: Refusing update from remote peer 9ca20d03820c40e6a7fe2efadf6ef182: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:31:23.038179 2741 consensus_queue.cc:1035] T 625893e548e14bd5b2dbe1589c7413ef P 9ca20d03820c40e6a7fe2efadf6ef182 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:31:23.038540 2736 consensus_queue.cc:1035] T 625893e548e14bd5b2dbe1589c7413ef P 9ca20d03820c40e6a7fe2efadf6ef182 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:31:23.065987 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:23.060365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41599 (local address 127.0.97.2:45485)
0926 03:31:23.061077 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:23.061095 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:23.061198 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:23.061429 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:23.061432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:23.061440 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:31:23.061666 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:23.061673 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:23.062733 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:23.062739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:23.064095 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:23.064108 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:23.064879 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:23.064885 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:23.065063 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:23.065120 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:31:23.065457 (+ 337us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:31:23.065471 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 03:31:23.065533 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 03:31:23.065622 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":476,"threads_started":1}
I20260926 03:31:23.065996 2744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:23.060423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45485 (local address 127.0.97.3:41599)
0926 03:31:23.060631 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:23.060641 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:23.061168 (+ 527us) server_negotiation.cc:407] Connection header received
0926 03:31:23.061276 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:23.061279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:23.061335 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:23.061402 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:31:23.061780 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:23.062613 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:23.064226 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:23.064712 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:23.064847 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:23.065275 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:31:23.065365 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:31:23.065444 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:23.065575 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 03:31:23.065628 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 03:31:23.093935 2128 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:31:23.094180 2128 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:31:23.097641 2128 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 726f37f70bde44e699103131c029838d: Sending DeleteTablet for 3 replicas of tablet 625893e548e14bd5b2dbe1589c7413ef
I20260926 03:31:23.098366 2271 tablet_service.cc:1514] Processing DeleteTablet for tablet 625893e548e14bd5b2dbe1589c7413ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:31:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55762
I20260926 03:31:23.098554 2405 tablet_service.cc:1514] Processing DeleteTablet for tablet 625893e548e14bd5b2dbe1589c7413ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:31:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58674
I20260926 03:31:23.098644 2718 tablet_replica.cc:330] T 625893e548e14bd5b2dbe1589c7413ef P 7f20b7a1aaba41a6b2c0c190047f7f98: stopping tablet replica
I20260926 03:31:23.098831 2718 raft_consensus.cc:2229] T 625893e548e14bd5b2dbe1589c7413ef P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:31:23.099010 2718 raft_consensus.cc:2258] T 625893e548e14bd5b2dbe1589c7413ef P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:31:23.099076 2544 tablet_service.cc:1514] Processing DeleteTablet for tablet 625893e548e14bd5b2dbe1589c7413ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:31:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39320
I20260926 03:31:23.099359 2719 tablet_replica.cc:330] T 625893e548e14bd5b2dbe1589c7413ef P 7ef0f75e0d754f0597ef751b629233dc: stopping tablet replica
I20260926 03:31:23.099557 2719 raft_consensus.cc:2229] T 625893e548e14bd5b2dbe1589c7413ef P 7ef0f75e0d754f0597ef751b629233dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:31:23.099735 2719 raft_consensus.cc:2258] T 625893e548e14bd5b2dbe1589c7413ef P 7ef0f75e0d754f0597ef751b629233dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:31:23.099565 2128 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34982:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 03:31:23.099978 2720 tablet_replica.cc:330] T 625893e548e14bd5b2dbe1589c7413ef P 9ca20d03820c40e6a7fe2efadf6ef182: stopping tablet replica
W20260926 03:31:23.100175 2128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:31:23.100164 2720 raft_consensus.cc:2229] T 625893e548e14bd5b2dbe1589c7413ef P 9ca20d03820c40e6a7fe2efadf6ef182 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:31:23.100195 2718 ts_tablet_manager.cc:1905] T 625893e548e14bd5b2dbe1589c7413ef P 7f20b7a1aaba41a6b2c0c190047f7f98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:31:23.100706 2720 raft_consensus.cc:2258] T 625893e548e14bd5b2dbe1589c7413ef P 9ca20d03820c40e6a7fe2efadf6ef182 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:31:23.102015 2719 ts_tablet_manager.cc:1905] T 625893e548e14bd5b2dbe1589c7413ef P 7ef0f75e0d754f0597ef751b629233dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:31:23.107530 2271 tablet_service.cc:1467] Processing CreateTablet for tablet 08c9bb082e2548749a98543d86c03ba7 (DEFAULT_TABLE table=test-table [id=22bc3838f0bb4e94bdbd0528b0098b10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:31:23.107926 2271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08c9bb082e2548749a98543d86c03ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:23.107857 2544 tablet_service.cc:1467] Processing CreateTablet for tablet 08c9bb082e2548749a98543d86c03ba7 (DEFAULT_TABLE table=test-table [id=22bc3838f0bb4e94bdbd0528b0098b10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:31:23.108235 2544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08c9bb082e2548749a98543d86c03ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:23.108879 2719 ts_tablet_manager.cc:1918] T 625893e548e14bd5b2dbe1589c7413ef P 7ef0f75e0d754f0597ef751b629233dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:31:23.109032 2719 log.cc:1196] T 625893e548e14bd5b2dbe1589c7413ef P 7ef0f75e0d754f0597ef751b629233dc: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-1/wal/wals/625893e548e14bd5b2dbe1589c7413ef
I20260926 03:31:23.109440 2719 ts_tablet_manager.cc:1939] T 625893e548e14bd5b2dbe1589c7413ef P 7ef0f75e0d754f0597ef751b629233dc: Deleting consensus metadata
I20260926 03:31:23.110854 2718 ts_tablet_manager.cc:1918] T 625893e548e14bd5b2dbe1589c7413ef P 7f20b7a1aaba41a6b2c0c190047f7f98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:31:23.110975 2720 ts_tablet_manager.cc:1905] T 625893e548e14bd5b2dbe1589c7413ef P 9ca20d03820c40e6a7fe2efadf6ef182: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:31:23.111009 2718 log.cc:1196] T 625893e548e14bd5b2dbe1589c7413ef P 7f20b7a1aaba41a6b2c0c190047f7f98: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-0/wal/wals/625893e548e14bd5b2dbe1589c7413ef
I20260926 03:31:23.111436 2718 ts_tablet_manager.cc:1939] T 625893e548e14bd5b2dbe1589c7413ef P 7f20b7a1aaba41a6b2c0c190047f7f98: Deleting consensus metadata
I20260926 03:31:23.113098 2115 catalog_manager.cc:4841] TS 7ef0f75e0d754f0597ef751b629233dc (127.0.97.2:41135): tablet 625893e548e14bd5b2dbe1589c7413ef (table test-table [id=71e10a6254884bc78c3ae418764084b7]) successfully deleted
I20260926 03:31:23.113243 2405 tablet_service.cc:1467] Processing CreateTablet for tablet 08c9bb082e2548749a98543d86c03ba7 (DEFAULT_TABLE table=test-table [id=22bc3838f0bb4e94bdbd0528b0098b10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:31:23.113653 2405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08c9bb082e2548749a98543d86c03ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:23.116737 2729 tablet_bootstrap.cc:492] T 08c9bb082e2548749a98543d86c03ba7 P 7f20b7a1aaba41a6b2c0c190047f7f98: Bootstrap starting.
I20260926 03:31:23.116701 2114 catalog_manager.cc:4841] TS 7f20b7a1aaba41a6b2c0c190047f7f98 (127.0.97.1:46239): tablet 625893e548e14bd5b2dbe1589c7413ef (table test-table [id=71e10a6254884bc78c3ae418764084b7]) successfully deleted
I20260926 03:31:23.117166 2730 tablet_bootstrap.cc:492] T 08c9bb082e2548749a98543d86c03ba7 P 9ca20d03820c40e6a7fe2efadf6ef182: Bootstrap starting.
I20260926 03:31:23.118315 2729 tablet_bootstrap.cc:654] T 08c9bb082e2548749a98543d86c03ba7 P 7f20b7a1aaba41a6b2c0c190047f7f98: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:23.118758 2730 tablet_bootstrap.cc:654] T 08c9bb082e2548749a98543d86c03ba7 P 9ca20d03820c40e6a7fe2efadf6ef182: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:23.119419 2731 tablet_bootstrap.cc:492] T 08c9bb082e2548749a98543d86c03ba7 P 7ef0f75e0d754f0597ef751b629233dc: Bootstrap starting.
I20260926 03:31:23.121050 2731 tablet_bootstrap.cc:654] T 08c9bb082e2548749a98543d86c03ba7 P 7ef0f75e0d754f0597ef751b629233dc: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:23.121253 2729 tablet_bootstrap.cc:492] T 08c9bb082e2548749a98543d86c03ba7 P 7f20b7a1aaba41a6b2c0c190047f7f98: No bootstrap required, opened a new log
I20260926 03:31:23.121399 2729 ts_tablet_manager.cc:1397] T 08c9bb082e2548749a98543d86c03ba7 P 7f20b7a1aaba41a6b2c0c190047f7f98: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260926 03:31:23.122008 2720 ts_tablet_manager.cc:1918] T 625893e548e14bd5b2dbe1589c7413ef P 9ca20d03820c40e6a7fe2efadf6ef182: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:31:23.122224 2720 log.cc:1196] T 625893e548e14bd5b2dbe1589c7413ef P 9ca20d03820c40e6a7fe2efadf6ef182: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-2/wal/wals/625893e548e14bd5b2dbe1589c7413ef
I20260926 03:31:23.122123 2729 raft_consensus.cc:348] T 08c9bb082e2548749a98543d86c03ba7 P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:23.122411 2729 raft_consensus.cc:374] T 08c9bb082e2548749a98543d86c03ba7 P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:23.122519 2729 raft_consensus.cc:729] T 08c9bb082e2548749a98543d86c03ba7 P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f20b7a1aaba41a6b2c0c190047f7f98, State: Initialized, Role: FOLLOWER
I20260926 03:31:23.122912 2720 ts_tablet_manager.cc:1939] T 625893e548e14bd5b2dbe1589c7413ef P 9ca20d03820c40e6a7fe2efadf6ef182: Deleting consensus metadata
I20260926 03:31:23.122779 2729 consensus_queue.cc:260] T 08c9bb082e2548749a98543d86c03ba7 P 7f20b7a1aaba41a6b2c0c190047f7f98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:23.123023 2730 tablet_bootstrap.cc:492] T 08c9bb082e2548749a98543d86c03ba7 P 9ca20d03820c40e6a7fe2efadf6ef182: No bootstrap required, opened a new log
I20260926 03:31:23.123231 2730 ts_tablet_manager.cc:1397] T 08c9bb082e2548749a98543d86c03ba7 P 9ca20d03820c40e6a7fe2efadf6ef182: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260926 03:31:23.124166 2115 catalog_manager.cc:4841] TS 9ca20d03820c40e6a7fe2efadf6ef182 (127.0.97.3:41599): tablet 625893e548e14bd5b2dbe1589c7413ef (table test-table [id=71e10a6254884bc78c3ae418764084b7]) successfully deleted
I20260926 03:31:23.124099 2730 raft_consensus.cc:348] T 08c9bb082e2548749a98543d86c03ba7 P 9ca20d03820c40e6a7fe2efadf6ef182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:23.124325 2730 raft_consensus.cc:374] T 08c9bb082e2548749a98543d86c03ba7 P 9ca20d03820c40e6a7fe2efadf6ef182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:23.124414 2730 raft_consensus.cc:729] T 08c9bb082e2548749a98543d86c03ba7 P 9ca20d03820c40e6a7fe2efadf6ef182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ca20d03820c40e6a7fe2efadf6ef182, State: Initialized, Role: FOLLOWER
I20260926 03:31:23.124683 2730 consensus_queue.cc:260] T 08c9bb082e2548749a98543d86c03ba7 P 9ca20d03820c40e6a7fe2efadf6ef182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:23.125250 2730 ts_tablet_manager.cc:1428] T 08c9bb082e2548749a98543d86c03ba7 P 9ca20d03820c40e6a7fe2efadf6ef182: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 03:31:23.125857 2729 ts_tablet_manager.cc:1428] T 08c9bb082e2548749a98543d86c03ba7 P 7f20b7a1aaba41a6b2c0c190047f7f98: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260926 03:31:23.126658 2731 tablet_bootstrap.cc:492] T 08c9bb082e2548749a98543d86c03ba7 P 7ef0f75e0d754f0597ef751b629233dc: No bootstrap required, opened a new log
I20260926 03:31:23.126816 2731 ts_tablet_manager.cc:1397] T 08c9bb082e2548749a98543d86c03ba7 P 7ef0f75e0d754f0597ef751b629233dc: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 03:31:23.127506 2731 raft_consensus.cc:348] T 08c9bb082e2548749a98543d86c03ba7 P 7ef0f75e0d754f0597ef751b629233dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:23.127744 2731 raft_consensus.cc:374] T 08c9bb082e2548749a98543d86c03ba7 P 7ef0f75e0d754f0597ef751b629233dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:23.127822 2731 raft_consensus.cc:729] T 08c9bb082e2548749a98543d86c03ba7 P 7ef0f75e0d754f0597ef751b629233dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef0f75e0d754f0597ef751b629233dc, State: Initialized, Role: FOLLOWER
I20260926 03:31:23.128490 2731 consensus_queue.cc:260] T 08c9bb082e2548749a98543d86c03ba7 P 7ef0f75e0d754f0597ef751b629233dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:23.129048 2731 ts_tablet_manager.cc:1428] T 08c9bb082e2548749a98543d86c03ba7 P 7ef0f75e0d754f0597ef751b629233dc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 03:31:23.169456 2705 raft_consensus.cc:482] T 08c9bb082e2548749a98543d86c03ba7 P 7ef0f75e0d754f0597ef751b629233dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:31:23.169677 2705 raft_consensus.cc:504] T 08c9bb082e2548749a98543d86c03ba7 P 7ef0f75e0d754f0597ef751b629233dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:23.170413 2705 leader_election.cc:288] T 08c9bb082e2548749a98543d86c03ba7 P 7ef0f75e0d754f0597ef751b629233dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f20b7a1aaba41a6b2c0c190047f7f98 (127.0.97.1:46239), 9ca20d03820c40e6a7fe2efadf6ef182 (127.0.97.3:41599)
I20260926 03:31:23.170989 2569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08c9bb082e2548749a98543d86c03ba7" candidate_uuid: "7ef0f75e0d754f0597ef751b629233dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" is_pre_election: true
I20260926 03:31:23.170967 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08c9bb082e2548749a98543d86c03ba7" candidate_uuid: "7ef0f75e0d754f0597ef751b629233dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" is_pre_election: true
I20260926 03:31:23.171222 2569 raft_consensus.cc:2454] T 08c9bb082e2548749a98543d86c03ba7 P 9ca20d03820c40e6a7fe2efadf6ef182 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ef0f75e0d754f0597ef751b629233dc in term 0.
I20260926 03:31:23.171217 2291 raft_consensus.cc:2454] T 08c9bb082e2548749a98543d86c03ba7 P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ef0f75e0d754f0597ef751b629233dc in term 0.
I20260926 03:31:23.171656 2357 leader_election.cc:302] T 08c9bb082e2548749a98543d86c03ba7 P 7ef0f75e0d754f0597ef751b629233dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ef0f75e0d754f0597ef751b629233dc, 7f20b7a1aaba41a6b2c0c190047f7f98; no voters:
I20260926 03:31:23.171962 2705 raft_consensus.cc:2781] T 08c9bb082e2548749a98543d86c03ba7 P 7ef0f75e0d754f0597ef751b629233dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:31:23.172087 2705 raft_consensus.cc:482] T 08c9bb082e2548749a98543d86c03ba7 P 7ef0f75e0d754f0597ef751b629233dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:31:23.172178 2705 raft_consensus.cc:3037] T 08c9bb082e2548749a98543d86c03ba7 P 7ef0f75e0d754f0597ef751b629233dc [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:23.175244 2705 raft_consensus.cc:504] T 08c9bb082e2548749a98543d86c03ba7 P 7ef0f75e0d754f0597ef751b629233dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:23.175846 2705 leader_election.cc:288] T 08c9bb082e2548749a98543d86c03ba7 P 7ef0f75e0d754f0597ef751b629233dc [CANDIDATE]: Term 1 election: Requested vote from peers 7f20b7a1aaba41a6b2c0c190047f7f98 (127.0.97.1:46239), 9ca20d03820c40e6a7fe2efadf6ef182 (127.0.97.3:41599)
I20260926 03:31:23.176273 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08c9bb082e2548749a98543d86c03ba7" candidate_uuid: "7ef0f75e0d754f0597ef751b629233dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98"
I20260926 03:31:23.176309 2569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08c9bb082e2548749a98543d86c03ba7" candidate_uuid: "7ef0f75e0d754f0597ef751b629233dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ca20d03820c40e6a7fe2efadf6ef182"
I20260926 03:31:23.176476 2291 raft_consensus.cc:3037] T 08c9bb082e2548749a98543d86c03ba7 P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:23.176555 2569 raft_consensus.cc:3037] T 08c9bb082e2548749a98543d86c03ba7 P 9ca20d03820c40e6a7fe2efadf6ef182 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:23.178576 2569 raft_consensus.cc:2454] T 08c9bb082e2548749a98543d86c03ba7 P 9ca20d03820c40e6a7fe2efadf6ef182 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ef0f75e0d754f0597ef751b629233dc in term 1.
I20260926 03:31:23.179001 2358 leader_election.cc:302] T 08c9bb082e2548749a98543d86c03ba7 P 7ef0f75e0d754f0597ef751b629233dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ef0f75e0d754f0597ef751b629233dc, 9ca20d03820c40e6a7fe2efadf6ef182; no voters:
I20260926 03:31:23.179275 2705 raft_consensus.cc:2781] T 08c9bb082e2548749a98543d86c03ba7 P 7ef0f75e0d754f0597ef751b629233dc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:31:23.179352 2291 raft_consensus.cc:2454] T 08c9bb082e2548749a98543d86c03ba7 P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ef0f75e0d754f0597ef751b629233dc in term 1.
I20260926 03:31:23.180006 2705 raft_consensus.cc:686] T 08c9bb082e2548749a98543d86c03ba7 P 7ef0f75e0d754f0597ef751b629233dc [term 1 LEADER]: Becoming Leader. State: Replica: 7ef0f75e0d754f0597ef751b629233dc, State: Running, Role: LEADER
I20260926 03:31:23.180263 2705 consensus_queue.cc:237] T 08c9bb082e2548749a98543d86c03ba7 P 7ef0f75e0d754f0597ef751b629233dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } }
I20260926 03:31:23.182865 2128 catalog_manager.cc:5495] T 08c9bb082e2548749a98543d86c03ba7 P 7ef0f75e0d754f0597ef751b629233dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ef0f75e0d754f0597ef751b629233dc (127.0.97.2). New cstate: current_term: 1 leader_uuid: "7ef0f75e0d754f0597ef751b629233dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ef0f75e0d754f0597ef751b629233dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 } health_report { overall_health: UNKNOWN } } }
I20260926 03:31:23.205663 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:23.200258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59930 (local address 127.0.97.2:41135)
0926 03:31:23.200416 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:23.200420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:23.200441 (+ 21us) server_negotiation.cc:407] Connection header received
0926 03:31:23.200488 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:23.200491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:23.200547 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:23.200673 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:31:23.201077 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:23.201827 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:23.203782 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:23.204239 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:23.204345 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:23.204884 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:31:23.205088 (+ 204us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:31:23.205162 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:23.205289 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 03:31:23.205356 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 03:31:23.214799 2291 raft_consensus.cc:1261] T 08c9bb082e2548749a98543d86c03ba7 P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 1 FOLLOWER]: Refusing update from remote peer 7ef0f75e0d754f0597ef751b629233dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:31:23.215003 2569 raft_consensus.cc:1261] T 08c9bb082e2548749a98543d86c03ba7 P 9ca20d03820c40e6a7fe2efadf6ef182 [term 1 FOLLOWER]: Refusing update from remote peer 7ef0f75e0d754f0597ef751b629233dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:31:23.215338 2687 consensus_queue.cc:1035] T 08c9bb082e2548749a98543d86c03ba7 P 7ef0f75e0d754f0597ef751b629233dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f20b7a1aaba41a6b2c0c190047f7f98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:31:23.216693 2687 consensus_queue.cc:1035] T 08c9bb082e2548749a98543d86c03ba7 P 7ef0f75e0d754f0597ef751b629233dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ca20d03820c40e6a7fe2efadf6ef182" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 03:31:23.235775 2698 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 03:31:24.818552 2128 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:31:24.818789 2128 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:31:24.822745 2128 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 726f37f70bde44e699103131c029838d: Sending DeleteTablet for 3 replicas of tablet 08c9bb082e2548749a98543d86c03ba7
I20260926 03:31:24.823594 2271 tablet_service.cc:1514] Processing DeleteTablet for tablet 08c9bb082e2548749a98543d86c03ba7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:31:24 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55762
I20260926 03:31:24.823710 2406 tablet_service.cc:1514] Processing DeleteTablet for tablet 08c9bb082e2548749a98543d86c03ba7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:31:24 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58674
I20260926 03:31:24.823722 2544 tablet_service.cc:1514] Processing DeleteTablet for tablet 08c9bb082e2548749a98543d86c03ba7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:31:24 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39320
I20260926 03:31:24.824301 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 2201
I20260926 03:31:24.829030 2783 tablet_replica.cc:330] T 08c9bb082e2548749a98543d86c03ba7 P 9ca20d03820c40e6a7fe2efadf6ef182: stopping tablet replica
I20260926 03:31:24.829326 2783 raft_consensus.cc:2229] T 08c9bb082e2548749a98543d86c03ba7 P 9ca20d03820c40e6a7fe2efadf6ef182 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:31:24.829607 2783 raft_consensus.cc:2258] T 08c9bb082e2548749a98543d86c03ba7 P 9ca20d03820c40e6a7fe2efadf6ef182 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:31:24.831030 2782 tablet_replica.cc:330] T 08c9bb082e2548749a98543d86c03ba7 P 7ef0f75e0d754f0597ef751b629233dc: stopping tablet replica
I20260926 03:31:24.831316 2782 raft_consensus.cc:2229] T 08c9bb082e2548749a98543d86c03ba7 P 7ef0f75e0d754f0597ef751b629233dc [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:31:24.831848 2782 raft_consensus.cc:2258] T 08c9bb082e2548749a98543d86c03ba7 P 7ef0f75e0d754f0597ef751b629233dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:31:24.831979 2783 ts_tablet_manager.cc:1905] T 08c9bb082e2548749a98543d86c03ba7 P 9ca20d03820c40e6a7fe2efadf6ef182: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:31:24.832892 2782 ts_tablet_manager.cc:1905] T 08c9bb082e2548749a98543d86c03ba7 P 7ef0f75e0d754f0597ef751b629233dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:31:24.836282 2781 tablet_replica.cc:330] T 08c9bb082e2548749a98543d86c03ba7 P 7f20b7a1aaba41a6b2c0c190047f7f98: stopping tablet replica
I20260926 03:31:24.837005 2781 raft_consensus.cc:2229] T 08c9bb082e2548749a98543d86c03ba7 P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:31:24.837311 2781 raft_consensus.cc:2258] T 08c9bb082e2548749a98543d86c03ba7 P 7f20b7a1aaba41a6b2c0c190047f7f98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:31:24.838317 2781 ts_tablet_manager.cc:1905] T 08c9bb082e2548749a98543d86c03ba7 P 7f20b7a1aaba41a6b2c0c190047f7f98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:31:24.840183 2782 ts_tablet_manager.cc:1918] T 08c9bb082e2548749a98543d86c03ba7 P 7ef0f75e0d754f0597ef751b629233dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 03:31:24.840315 2782 log.cc:1196] T 08c9bb082e2548749a98543d86c03ba7 P 7ef0f75e0d754f0597ef751b629233dc: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-1/wal/wals/08c9bb082e2548749a98543d86c03ba7
I20260926 03:31:24.840735 2783 ts_tablet_manager.cc:1918] T 08c9bb082e2548749a98543d86c03ba7 P 9ca20d03820c40e6a7fe2efadf6ef182: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 03:31:24.840797 2782 ts_tablet_manager.cc:1939] T 08c9bb082e2548749a98543d86c03ba7 P 7ef0f75e0d754f0597ef751b629233dc: Deleting consensus metadata
I20260926 03:31:24.840884 2783 log.cc:1196] T 08c9bb082e2548749a98543d86c03ba7 P 9ca20d03820c40e6a7fe2efadf6ef182: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-2/wal/wals/08c9bb082e2548749a98543d86c03ba7
I20260926 03:31:24.841257 2783 ts_tablet_manager.cc:1939] T 08c9bb082e2548749a98543d86c03ba7 P 9ca20d03820c40e6a7fe2efadf6ef182: Deleting consensus metadata
I20260926 03:31:24.841910 2115 catalog_manager.cc:4841] TS 7ef0f75e0d754f0597ef751b629233dc (127.0.97.2:41135): tablet 08c9bb082e2548749a98543d86c03ba7 (table test-table [id=22bc3838f0bb4e94bdbd0528b0098b10]) successfully deleted
I20260926 03:31:24.842981 2115 catalog_manager.cc:4841] TS 9ca20d03820c40e6a7fe2efadf6ef182 (127.0.97.3:41599): tablet 08c9bb082e2548749a98543d86c03ba7 (table test-table [id=22bc3838f0bb4e94bdbd0528b0098b10]) successfully deleted
I20260926 03:31:24.844251 2781 ts_tablet_manager.cc:1918] T 08c9bb082e2548749a98543d86c03ba7 P 7f20b7a1aaba41a6b2c0c190047f7f98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 03:31:24.844348 2781 log.cc:1196] T 08c9bb082e2548749a98543d86c03ba7 P 7f20b7a1aaba41a6b2c0c190047f7f98: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790393463219689-388-0/minicluster-data/ts-0/wal/wals/08c9bb082e2548749a98543d86c03ba7
I20260926 03:31:24.844749 2781 ts_tablet_manager.cc:1939] T 08c9bb082e2548749a98543d86c03ba7 P 7f20b7a1aaba41a6b2c0c190047f7f98: Deleting consensus metadata
I20260926 03:31:24.845755 2114 catalog_manager.cc:4841] TS 7f20b7a1aaba41a6b2c0c190047f7f98 (127.0.97.1:46239): tablet 08c9bb082e2548749a98543d86c03ba7 (table test-table [id=22bc3838f0bb4e94bdbd0528b0098b10]) successfully deleted
Sep 26 03:31:24 dist-test-slave-dt7p krb5kdc[2085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393482, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:31:24.854333 2785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:24.825679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33020 (local address 127.0.97.1:46239)
0926 03:31:24.830317 (+ 4638us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:24.830327 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:24.830976 (+ 649us) server_negotiation.cc:407] Connection header received
0926 03:31:24.831052 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:24.831060 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:24.831147 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:24.831251 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:24.833268 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:24.834404 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:24.836254 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:24.837385 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:24.837515 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:24.849106 (+ 11591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:24.849123 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:24.849141 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:24.849184 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:24.851893 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:24.852514 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:24.852521 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:24.852524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:24.852719 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:24.853103 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:24.853106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:24.853108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:24.853452 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:24.853671 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:24.853925 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 03:31:24.854090 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4446,"thread_start_us":4316,"threads_started":1}
W20260926 03:31:24.855403 2332 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33020
W20260926 03:31:24.856022 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 2201: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:31:24.856202 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 2201
I20260926 03:31:24.874411 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 2341
Sep 26 03:31:24 dist-test-slave-dt7p krb5kdc[2085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393482, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:31:24.889075 2786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:24.875187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59940 (local address 127.0.97.2:41135)
0926 03:31:24.875739 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:24.875743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:24.875760 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:31:24.875814 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:24.875818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:24.875874 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:24.875971 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:24.877232 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:24.877988 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:24.879205 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:24.879600 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:24.879701 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:24.883746 (+ 4045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:24.883762 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:24.883779 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:24.883816 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:24.886764 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:24.887345 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:24.887352 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:24.887355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:24.887484 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:24.887775 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:24.887778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:24.887779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:24.888144 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:24.888368 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:24.888625 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 03:31:24.888785 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":311,"threads_started":1}
W20260926 03:31:24.889792 2470 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59940
W20260926 03:31:24.890486 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 2341: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:31:24.890642 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 2341
I20260926 03:31:24.916364 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 2480
Sep 26 03:31:24 dist-test-slave-dt7p krb5kdc[2085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393482, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:31:24.931721 2787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:24.917188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39120 (local address 127.0.97.3:41599)
0926 03:31:24.917672 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:24.917676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:24.917693 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:31:24.917746 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:24.917749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:24.917806 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:24.917898 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:24.919156 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:24.919877 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:24.921296 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:24.921789 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:24.921892 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:24.926173 (+ 4281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:24.926190 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:24.926207 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:24.926241 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:24.929064 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:24.929648 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:24.929656 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:24.929659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:24.929780 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:24.930096 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:24.930100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:24.930102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:24.930453 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:24.930759 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:24.931165 (+ 406us) server_negotiation.cc:299] Negotiation successful
0926 03:31:24.931327 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":266,"threads_started":1}
W20260926 03:31:24.932360 2610 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39120
W20260926 03:31:24.933108 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 2480: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:31:24.933303 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 2480
I20260926 03:31:24.955160 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 2098
I20260926 03:31:24.956435 2170 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:31:25.071445 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 2098
2026-09-26T03:31:25Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8402 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:31:25 dist-test-slave-dt7p krb5kdc[2791](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:31:25 dist-test-slave-dt7p krb5kdc[2791](info): set up 2 sockets
Sep 26 03:31:25 dist-test-slave-dt7p krb5kdc[2791](info): commencing operation
krb5kdc: starting...
W20260926 03:31:26.207854 388 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.004s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260926 03:31:26.332543 388 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.124s user 0.000s sys 0.005s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260926 03:31:26.562139 388 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.229s user 0.002s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260926 03:31:26.673836 388 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.111s user 0.000s sys 0.005s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/krb5kdc/test-user.keytab.
W20260926 03:31:27.439278 388 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.765s user 0.000s sys 0.005s
Sep 26 03:31:27 dist-test-slave-dt7p krb5kdc[2791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:31:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:31:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
W20260926 03:31:27.659792 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.62: real 0.168s user 0.003s sys 0.007s
I20260926 03:31:27.660907 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35547
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:27.979748 2804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:27.980036 2804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:27.980122 2804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:27.987280 2804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:31:27.987377 2804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:27.987421 2804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:27.987468 2804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:31:27.987511 2804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:31:27.997120 2804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:27.998373 2804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:27.999627 2804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:28.011463 2810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:28.011473 2809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:28.011461 2812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:28.011875 2804 server_base.cc:1034] running on GCE node
I20260926 03:31:28.012861 2804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:28.014153 2804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:28.015348 2804 hybrid_clock.cc:648] HybridClock initialized: now 1790393488015306 us; error 28 us; skew 500 ppm
I20260926 03:31:28.015686 2804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:28 dist-test-slave-dt7p krb5kdc[2791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:28.018853 2804 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:31:28.020409 2804 webserver.cc:460] Webserver started at http://127.0.97.62:42607/ using document root <none> and password file <none>
I20260926 03:31:28.021147 2804 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:28.021277 2804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:28.021620 2804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:28.024524 2804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "91c6352814934fc099bb863516caaf69"
format_stamp: "Formatted at 2026-09-26 03:31:28 on dist-test-slave-dt7p"
server_key: "d0c8067f2ff9458a3ec8308d5342852e"
server_key_iv: "9cd0f2cd60582bf2bd517cba93b2eea9"
server_key_version: "encryptionkey@0"
I20260926 03:31:28.025382 2804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "91c6352814934fc099bb863516caaf69"
format_stamp: "Formatted at 2026-09-26 03:31:28 on dist-test-slave-dt7p"
server_key: "d0c8067f2ff9458a3ec8308d5342852e"
server_key_iv: "9cd0f2cd60582bf2bd517cba93b2eea9"
server_key_version: "encryptionkey@0"
I20260926 03:31:28.030522 2804 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:31:28.033910 2819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:28.035205 2804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:31:28.035387 2804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "91c6352814934fc099bb863516caaf69"
format_stamp: "Formatted at 2026-09-26 03:31:28 on dist-test-slave-dt7p"
server_key: "d0c8067f2ff9458a3ec8308d5342852e"
server_key_iv: "9cd0f2cd60582bf2bd517cba93b2eea9"
server_key_version: "encryptionkey@0"
I20260926 03:31:28.035573 2804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:28.046241 2804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:28.050107 2804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:28.050401 2804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:28.068339 2804 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35547
I20260926 03:31:28.068334 2870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35547 every 8 connection(s)
I20260926 03:31:28.069694 2804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:31:28.073381 2871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:28.078204 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 2804
I20260926 03:31:28.078411 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:31:28.079200 388 external_mini_cluster.cc:1428] Setting key fae22c5505d36fa014e21aa77968af04
I20260926 03:31:28.084656 2871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91c6352814934fc099bb863516caaf69: Bootstrap starting.
I20260926 03:31:28.088203 2871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91c6352814934fc099bb863516caaf69: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:28.089422 2871 log.cc:824] T 00000000000000000000000000000000 P 91c6352814934fc099bb863516caaf69: Log is configured to *not* fsync() on all Append() calls
I20260926 03:31:28.092079 2871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91c6352814934fc099bb863516caaf69: No bootstrap required, opened a new log
Sep 26 03:31:28 dist-test-slave-dt7p krb5kdc[2791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:31:28.099627 2871 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91c6352814934fc099bb863516caaf69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91c6352814934fc099bb863516caaf69" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35547 } }
I20260926 03:31:28.099989 2871 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91c6352814934fc099bb863516caaf69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:28.100121 2871 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91c6352814934fc099bb863516caaf69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91c6352814934fc099bb863516caaf69, State: Initialized, Role: FOLLOWER
I20260926 03:31:28.100811 2871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91c6352814934fc099bb863516caaf69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91c6352814934fc099bb863516caaf69" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35547 } }
I20260926 03:31:28.101082 2871 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91c6352814934fc099bb863516caaf69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:31:28.101241 2871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91c6352814934fc099bb863516caaf69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:31:28.101421 2871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91c6352814934fc099bb863516caaf69 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:28.101755 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:28.081190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46556 (local address 127.0.97.62:35547)
0926 03:31:28.082014 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:28.082025 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:28.082069 (+ 44us) server_negotiation.cc:407] Connection header received
0926 03:31:28.083685 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:28.083704 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:28.084048 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:28.084351 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:28.086149 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:28.087453 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:28.089288 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:28.090128 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:28.090316 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:28.094318 (+ 4002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:28.094326 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:28.094339 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:28.094373 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:28.097523 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:28.098206 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:28.098213 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:28.098221 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:28.098389 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:28.098781 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:28.098790 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:28.098792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:28.099299 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:28.099640 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:28.100046 (+ 406us) server_negotiation.cc:299] Negotiation successful
0926 03:31:28.100531 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":408,"threads_started":1}
I20260926 03:31:28.104202 2871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91c6352814934fc099bb863516caaf69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91c6352814934fc099bb863516caaf69" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35547 } }
I20260926 03:31:28.104830 2871 leader_election.cc:302] T 00000000000000000000000000000000 P 91c6352814934fc099bb863516caaf69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91c6352814934fc099bb863516caaf69; no voters:
I20260926 03:31:28.105419 2871 leader_election.cc:288] T 00000000000000000000000000000000 P 91c6352814934fc099bb863516caaf69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:31:28.105554 2876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91c6352814934fc099bb863516caaf69 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:31:28.106318 2876 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91c6352814934fc099bb863516caaf69 [term 1 LEADER]: Becoming Leader. State: Replica: 91c6352814934fc099bb863516caaf69, State: Running, Role: LEADER
I20260926 03:31:28.106860 2876 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91c6352814934fc099bb863516caaf69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91c6352814934fc099bb863516caaf69" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35547 } }
I20260926 03:31:28.107502 2871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91c6352814934fc099bb863516caaf69 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:31:28.111478 2878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91c6352814934fc099bb863516caaf69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91c6352814934fc099bb863516caaf69. Latest consensus state: current_term: 1 leader_uuid: "91c6352814934fc099bb863516caaf69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91c6352814934fc099bb863516caaf69" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35547 } } }
I20260926 03:31:28.112293 2878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91c6352814934fc099bb863516caaf69 [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:28.110044 2877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91c6352814934fc099bb863516caaf69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91c6352814934fc099bb863516caaf69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91c6352814934fc099bb863516caaf69" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35547 } } }
I20260926 03:31:28.113878 2877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91c6352814934fc099bb863516caaf69 [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:28.113931 2886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:31:28.119102 2886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:31:28.127285 2886 catalog_manager.cc:1347] Generated new cluster ID: 3781b570cd0448bfb6f5f1bfef83df85
I20260926 03:31:28.127391 2886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:31:28.141669 2886 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:31:28.143071 2886 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:31:28.152165 2886 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91c6352814934fc099bb863516caaf69: Generated new TSK 0
I20260926 03:31:28.153157 2886 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:31:28.204249 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35547
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:28.526018 2897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:28.526332 2897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:28.526489 2897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:28.533684 2897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:28.533785 2897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:28.533962 2897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:31:28.544191 2897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:28.545611 2897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:28.546908 2897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:28.559706 2902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:28.559767 2903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:28.559669 2905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:28.561512 2897 server_base.cc:1034] running on GCE node
I20260926 03:31:28.562156 2897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:28.563117 2897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:28.564373 2897 hybrid_clock.cc:648] HybridClock initialized: now 1790393488564328 us; error 48 us; skew 500 ppm
I20260926 03:31:28.564774 2897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:28 dist-test-slave-dt7p krb5kdc[2791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:28.568153 2897 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:31:28.569621 2897 webserver.cc:460] Webserver started at http://127.0.97.1:45995/ using document root <none> and password file <none>
I20260926 03:31:28.570338 2897 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:28.570489 2897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:28.570847 2897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:28.573941 2897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-0/data/instance:
uuid: "a1930efa40ef448494c47dd3fe1f0e5e"
format_stamp: "Formatted at 2026-09-26 03:31:28 on dist-test-slave-dt7p"
server_key: "4ecf0414f5c7a6fba630c7c95b01f381"
server_key_iv: "ab18abef8edb27a8529ba4311da6a8ad"
server_key_version: "encryptionkey@0"
I20260926 03:31:28.574769 2897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a1930efa40ef448494c47dd3fe1f0e5e"
format_stamp: "Formatted at 2026-09-26 03:31:28 on dist-test-slave-dt7p"
server_key: "4ecf0414f5c7a6fba630c7c95b01f381"
server_key_iv: "ab18abef8edb27a8529ba4311da6a8ad"
server_key_version: "encryptionkey@0"
I20260926 03:31:28.579907 2897 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 03:31:28.583513 2912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:28.584971 2897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:31:28.585155 2897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-0/wal
uuid: "a1930efa40ef448494c47dd3fe1f0e5e"
format_stamp: "Formatted at 2026-09-26 03:31:28 on dist-test-slave-dt7p"
server_key: "4ecf0414f5c7a6fba630c7c95b01f381"
server_key_iv: "ab18abef8edb27a8529ba4311da6a8ad"
server_key_version: "encryptionkey@0"
I20260926 03:31:28.585368 2897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:28.608433 2897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:28.613472 2897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:28.613879 2897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:28.614871 2897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:31:28.616204 2897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:31:28.616279 2897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:28.616364 2897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:31:28.616411 2897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:28.651332 2897 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33499
I20260926 03:31:28.651430 3024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33499 every 8 connection(s)
I20260926 03:31:28.652748 2897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:31:28.662621 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 2897
I20260926 03:31:28.662827 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:31:28.663440 388 external_mini_cluster.cc:1428] Setting key 64e52e3edfed8cd18c1aede3712bd9ab
Sep 26 03:31:28 dist-test-slave-dt7p krb5kdc[2791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:31:28.673545 3027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:28.655594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35233 (local address 127.0.97.62:35547)
0926 03:31:28.656019 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:28.656023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:28.657181 (+ 1158us) server_negotiation.cc:407] Connection header received
0926 03:31:28.659013 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:28.659019 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:28.659118 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:28.659250 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:28.660843 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:28.661584 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:28.663248 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:28.663709 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:28.663845 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:28.668062 (+ 4217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:28.668067 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:28.668071 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:28.668101 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:28.670546 (+ 2445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:28.671246 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:28.671251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:28.671255 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:28.671370 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:28.671861 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:28.671865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:28.671867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:28.672115 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:28.672227 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:28.673113 (+ 886us) server_negotiation.cc:299] Negotiation successful
0926 03:31:28.673274 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":205,"threads_started":1}
I20260926 03:31:28.675078 3028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:28.656191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35547 (local address 127.0.97.1:35233)
0926 03:31:28.657009 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:28.657060 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:28.658738 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:28.659484 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:28.659492 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:28.660015 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:28.660675 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:28.660694 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:28.661746 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:28.661751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:28.663094 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:28.663111 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:28.663933 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:28.663942 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:28.664175 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:28.665001 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:28.665053 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:28.667843 (+ 2790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:28.670686 (+ 2843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:28.670696 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:28.670713 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:28.671120 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:28.671477 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:28.671484 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:28.671490 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:28.671729 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:28.672240 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:28.672251 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:28.672719 (+ 468us) client_negotiation.cc:769] Sending connection context
0926 03:31:28.673017 (+ 298us) client_negotiation.cc:240] Negotiation successful
0926 03:31:28.673593 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":383,"threads_started":1}
I20260926 03:31:28.676687 3025 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35547
I20260926 03:31:28.677003 3025 heartbeater.cc:464] Registering TS with master...
I20260926 03:31:28.677695 3025 heartbeater.cc:511] Master 127.0.97.62:35547 requested a full tablet report, sending...
I20260926 03:31:28.679829 2836 ts_manager.cc:194] Registered new tserver with Master: a1930efa40ef448494c47dd3fe1f0e5e (127.0.97.1:33499)
I20260926 03:31:28.681329 2836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:35233
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:31:28.706185 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35547
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:29.023978 3031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:29.024335 3031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:29.024510 3031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:29.031947 3031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:29.032119 3031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:29.032305 3031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:31:29.043761 3031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:29.045315 3031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:29.046877 3031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:29.060397 3036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:29.060422 3039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:29.060500 3037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:29.061189 3031 server_base.cc:1034] running on GCE node
I20260926 03:31:29.061842 3031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:29.062772 3031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:29.063993 3031 hybrid_clock.cc:648] HybridClock initialized: now 1790393489063945 us; error 41 us; skew 500 ppm
I20260926 03:31:29.064335 3031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:29 dist-test-slave-dt7p krb5kdc[2791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:29.067622 3031 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:31:29.069494 3031 webserver.cc:460] Webserver started at http://127.0.97.2:42883/ using document root <none> and password file <none>
I20260926 03:31:29.070333 3031 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:29.070470 3031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:29.070778 3031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:29.074622 3031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-1/data/instance:
uuid: "ac293da20b0a46c29bc68d2453dcdba6"
format_stamp: "Formatted at 2026-09-26 03:31:29 on dist-test-slave-dt7p"
server_key: "fbbefd9a1e24670490af50f563382153"
server_key_iv: "c7c9acd0a72e17f061e98ae982f3a979"
server_key_version: "encryptionkey@0"
I20260926 03:31:29.075538 3031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ac293da20b0a46c29bc68d2453dcdba6"
format_stamp: "Formatted at 2026-09-26 03:31:29 on dist-test-slave-dt7p"
server_key: "fbbefd9a1e24670490af50f563382153"
server_key_iv: "c7c9acd0a72e17f061e98ae982f3a979"
server_key_version: "encryptionkey@0"
I20260926 03:31:29.081650 3031 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260926 03:31:29.085765 3046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:29.087235 3031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:31:29.087378 3031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-1/wal
uuid: "ac293da20b0a46c29bc68d2453dcdba6"
format_stamp: "Formatted at 2026-09-26 03:31:29 on dist-test-slave-dt7p"
server_key: "fbbefd9a1e24670490af50f563382153"
server_key_iv: "c7c9acd0a72e17f061e98ae982f3a979"
server_key_version: "encryptionkey@0"
I20260926 03:31:29.087579 3031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:29.106539 3031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:29.110677 3031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:29.110905 3031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:29.111797 3031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:31:29.113143 3031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:31:29.113214 3031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:29.113297 3031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:31:29.113353 3031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:29.145884 3031 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34967
I20260926 03:31:29.145923 3158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34967 every 8 connection(s)
I20260926 03:31:29.147183 3031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:31:29.153136 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 3031
I20260926 03:31:29.153316 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:31:29.153750 388 external_mini_cluster.cc:1428] Setting key d194d7b0340e4d2eba857adf49120b79
Sep 26 03:31:29 dist-test-slave-dt7p krb5kdc[2791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:31:29.169040 3027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:29.149772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41775 (local address 127.0.97.62:35547)
0926 03:31:29.149984 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:29.149990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:29.151046 (+ 1056us) server_negotiation.cc:407] Connection header received
0926 03:31:29.152837 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:29.152846 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:29.152952 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:29.153072 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:29.154870 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:29.155834 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:29.157423 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:29.158174 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:29.158311 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:29.162089 (+ 3778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:29.162098 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:29.162106 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:29.162150 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:29.165786 (+ 3636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:29.166495 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:29.166504 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:29.166511 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:29.166660 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:29.167142 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:29.167150 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:29.167157 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:29.167533 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:29.167686 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:29.168342 (+ 656us) server_negotiation.cc:299] Negotiation successful
0926 03:31:29.168621 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:31:29.170060 3161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:29.150213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35547 (local address 127.0.97.2:41775)
0926 03:31:29.150893 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:29.150930 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:29.152512 (+ 1582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:29.153277 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:29.153288 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:29.153873 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:29.154644 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:29.154674 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:29.155987 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:29.155991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:29.157283 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:29.157294 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:29.158381 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:29.158385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:29.158574 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:29.159272 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:29.159298 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:29.161862 (+ 2564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:29.165920 (+ 4058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:29.165927 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:29.165940 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:29.166357 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:29.166810 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:29.166813 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:29.166815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:29.167005 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:29.167653 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:29.167658 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:29.167956 (+ 298us) client_negotiation.cc:769] Sending connection context
0926 03:31:29.168189 (+ 233us) client_negotiation.cc:240] Negotiation successful
0926 03:31:29.168706 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":262,"threads_started":1}
I20260926 03:31:29.171615 3159 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35547
I20260926 03:31:29.171927 3159 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 03:31:29.172705 3159 heartbeater.cc:511] Master 127.0.97.62:35547 requested a full tablet report, sending...
I20260926 03:31:29.174098 2836 ts_manager.cc:194] Registered new tserver with Master: ac293da20b0a46c29bc68d2453dcdba6 (127.0.97.2:34967)
I20260926 03:31:29.174952 2836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41775
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:31:29.197515 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35547
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:29.517611 3164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:29.517897 3164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:29.518026 3164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:29.525079 3164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:29.525180 3164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:29.525290 3164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:31:29.534842 3164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:29.536031 3164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:29.537282 3164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:29.550915 3172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:29.551069 3170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:29.551045 3169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:29.551857 3164 server_base.cc:1034] running on GCE node
I20260926 03:31:29.552517 3164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:29.553535 3164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:29.554756 3164 hybrid_clock.cc:648] HybridClock initialized: now 1790393489554708 us; error 46 us; skew 500 ppm
I20260926 03:31:29.555079 3164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:29 dist-test-slave-dt7p krb5kdc[2791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:29.558362 3164 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:31:29.559844 3164 webserver.cc:460] Webserver started at http://127.0.97.3:41655/ using document root <none> and password file <none>
I20260926 03:31:29.560535 3164 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:29.560729 3164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:29.561039 3164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:29.563942 3164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-2/data/instance:
uuid: "bdde5eef497f40c7b2f30a3dda3dec68"
format_stamp: "Formatted at 2026-09-26 03:31:29 on dist-test-slave-dt7p"
server_key: "145ef39fd42ae442ee0ce66e79891985"
server_key_iv: "6014ee2b8f585ace8d34932029991cff"
server_key_version: "encryptionkey@0"
I20260926 03:31:29.564829 3164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-2/wal/instance:
uuid: "bdde5eef497f40c7b2f30a3dda3dec68"
format_stamp: "Formatted at 2026-09-26 03:31:29 on dist-test-slave-dt7p"
server_key: "145ef39fd42ae442ee0ce66e79891985"
server_key_iv: "6014ee2b8f585ace8d34932029991cff"
server_key_version: "encryptionkey@0"
I20260926 03:31:29.570259 3164 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:31:29.573733 3179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:29.575055 3164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:31:29.575224 3164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-2/wal
uuid: "bdde5eef497f40c7b2f30a3dda3dec68"
format_stamp: "Formatted at 2026-09-26 03:31:29 on dist-test-slave-dt7p"
server_key: "145ef39fd42ae442ee0ce66e79891985"
server_key_iv: "6014ee2b8f585ace8d34932029991cff"
server_key_version: "encryptionkey@0"
I20260926 03:31:29.575417 3164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:29.611617 3164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:29.615971 3164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:29.616264 3164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:29.617296 3164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:31:29.618618 3164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:31:29.618718 3164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:29.618866 3164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:31:29.618937 3164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:29.652136 3164 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38627
I20260926 03:31:29.652173 3291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38627 every 8 connection(s)
I20260926 03:31:29.653450 3164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:31:29.655237 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 3164
I20260926 03:31:29.655416 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:31:29.655843 388 external_mini_cluster.cc:1428] Setting key 3e74d9b5fe00ce68c426cc4453a333af
Sep 26 03:31:29 dist-test-slave-dt7p krb5kdc[2791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:31:29.674335 3027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:29.656888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50259 (local address 127.0.97.62:35547)
0926 03:31:29.657094 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:29.657100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:29.658477 (+ 1377us) server_negotiation.cc:407] Connection header received
0926 03:31:29.660308 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:29.660312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:29.660381 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:29.660458 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:29.661906 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:29.662674 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:29.664154 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:29.664709 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:29.664823 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:29.668042 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:29.668048 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:29.668051 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:29.668086 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:29.671166 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:29.671943 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:29.671950 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:29.671954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:29.672094 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:29.672542 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:29.672549 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:29.672553 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:29.672862 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:29.673028 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:29.673738 (+ 710us) server_negotiation.cc:299] Negotiation successful
0926 03:31:29.673960 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 03:31:29.675318 3294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:29.657388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35547 (local address 127.0.97.3:50259)
0926 03:31:29.658298 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:29.658357 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:29.660077 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:29.660729 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:29.660736 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:29.661221 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:29.661771 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:29.661787 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:29.662788 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:29.662791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:29.664005 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:29.664016 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:29.664905 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:29.664910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:29.665101 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:29.665788 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:29.665813 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:29.667850 (+ 2037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:29.671348 (+ 3498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:29.671358 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:29.671374 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:29.671811 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:29.672225 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:29.672230 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:29.672235 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:29.672422 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:29.673007 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:29.673015 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:29.673349 (+ 334us) client_negotiation.cc:769] Sending connection context
0926 03:31:29.673630 (+ 281us) client_negotiation.cc:240] Negotiation successful
0926 03:31:29.674172 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":364,"threads_started":1}
I20260926 03:31:29.676832 3292 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35547
I20260926 03:31:29.677135 3292 heartbeater.cc:464] Registering TS with master...
I20260926 03:31:29.677824 3292 heartbeater.cc:511] Master 127.0.97.62:35547 requested a full tablet report, sending...
I20260926 03:31:29.679193 2836 ts_manager.cc:194] Registered new tserver with Master: bdde5eef497f40c7b2f30a3dda3dec68 (127.0.97.3:38627)
I20260926 03:31:29.680040 2836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50259
I20260926 03:31:29.683578 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:31:29.684540 3025 heartbeater.cc:503] Master 127.0.97.62:35547 was elected leader, sending a full tablet report...
W20260926 03:31:29.713366 3027 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:31:29.710841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46562 (local address 127.0.97.62:35547)
0926 03:31:29.711013 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:29.711018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:29.711501 (+ 483us) server_negotiation.cc:407] Connection header received
0926 03:31:29.711708 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:29.711712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:29.711774 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:29.711877 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:29.713181 (+ 1304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46562: BlockingRecv error: recv got EOF from 127.0.0.1:46562 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 03:31:29.717032 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 2897
W20260926 03:31:29.719837 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 2897: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:33499: server requires authentication, but client does not have Kerberos credentials available
I20260926 03:31:29.720045 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 2897
I20260926 03:31:29.735668 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 3031
W20260926 03:31:29.738253 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 3031: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:34967: server requires authentication, but client does not have Kerberos credentials available
I20260926 03:31:29.738363 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 3031
I20260926 03:31:29.755086 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 3164
W20260926 03:31:29.758487 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 3164: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:38627: server requires authentication, but client does not have Kerberos credentials available
I20260926 03:31:29.758646 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 3164
I20260926 03:31:29.774260 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 2804
I20260926 03:31:29.775460 2866 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:31:29.885730 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 2804
2026-09-26T03:31:29Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4814 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:31:29 dist-test-slave-dt7p krb5kdc[3309](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:31:29 dist-test-slave-dt7p krb5kdc[3309](info): set up 2 sockets
Sep 26 03:31:29 dist-test-slave-dt7p krb5kdc[3309](info): commencing operation
krb5kdc: starting...
W20260926 03:31:30.988884 388 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:31:31 dist-test-slave-dt7p krb5kdc[3309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:31:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:31:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:31:31.180939 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44989
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:31.494625 3322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:31.494899 3322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:31.495081 3322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:31.502143 3322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:31:31.502238 3322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:31.502348 3322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:31.502467 3322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:31:31.502552 3322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:31:31.511940 3322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:31.513286 3322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:31.514591 3322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:31.526127 3330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:31.526165 3328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:31.526189 3327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:31.527045 3322 server_base.cc:1034] running on GCE node
I20260926 03:31:31.527799 3322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:31.529246 3322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:31.530476 3322 hybrid_clock.cc:648] HybridClock initialized: now 1790393491530440 us; error 54 us; skew 500 ppm
I20260926 03:31:31.530833 3322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:31 dist-test-slave-dt7p krb5kdc[3309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:31.534050 3322 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:31:31.535554 3322 webserver.cc:460] Webserver started at http://127.0.97.62:44567/ using document root <none> and password file <none>
I20260926 03:31:31.536299 3322 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:31.536443 3322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:31.536815 3322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:31.539667 3322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "c3eeb7d57204473aa34648dd2ec6d1d9"
format_stamp: "Formatted at 2026-09-26 03:31:31 on dist-test-slave-dt7p"
server_key: "15aa630231f531c86401a12e73f98500"
server_key_iv: "5f058c46b0d0a57c1017a38aefda21a1"
server_key_version: "encryptionkey@0"
I20260926 03:31:31.540421 3322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "c3eeb7d57204473aa34648dd2ec6d1d9"
format_stamp: "Formatted at 2026-09-26 03:31:31 on dist-test-slave-dt7p"
server_key: "15aa630231f531c86401a12e73f98500"
server_key_iv: "5f058c46b0d0a57c1017a38aefda21a1"
server_key_version: "encryptionkey@0"
I20260926 03:31:31.545418 3322 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:31:31.548906 3337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:31.550379 3322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:31:31.550570 3322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "c3eeb7d57204473aa34648dd2ec6d1d9"
format_stamp: "Formatted at 2026-09-26 03:31:31 on dist-test-slave-dt7p"
server_key: "15aa630231f531c86401a12e73f98500"
server_key_iv: "5f058c46b0d0a57c1017a38aefda21a1"
server_key_version: "encryptionkey@0"
I20260926 03:31:31.550769 3322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:31.595158 3322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:31.599187 3322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:31.599470 3322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:31.617363 3322 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44989
I20260926 03:31:31.617353 3388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44989 every 8 connection(s)
I20260926 03:31:31.618710 3322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:31:31.622262 3389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:31.627796 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 3322
I20260926 03:31:31.628023 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:31:31.628666 388 external_mini_cluster.cc:1428] Setting key 3f8049281bdf1be24e2b8b0459d3af2a
I20260926 03:31:31.633019 3389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3eeb7d57204473aa34648dd2ec6d1d9: Bootstrap starting.
I20260926 03:31:31.636777 3389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3eeb7d57204473aa34648dd2ec6d1d9: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:31.637895 3389 log.cc:824] T 00000000000000000000000000000000 P c3eeb7d57204473aa34648dd2ec6d1d9: Log is configured to *not* fsync() on all Append() calls
I20260926 03:31:31.641119 3389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3eeb7d57204473aa34648dd2ec6d1d9: No bootstrap required, opened a new log
Sep 26 03:31:31 dist-test-slave-dt7p krb5kdc[3309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:31:31.651662 3389 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3eeb7d57204473aa34648dd2ec6d1d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3eeb7d57204473aa34648dd2ec6d1d9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44989 } }
I20260926 03:31:31.652148 3389 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3eeb7d57204473aa34648dd2ec6d1d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:31.652249 3389 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3eeb7d57204473aa34648dd2ec6d1d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3eeb7d57204473aa34648dd2ec6d1d9, State: Initialized, Role: FOLLOWER
I20260926 03:31:31.652967 3389 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3eeb7d57204473aa34648dd2ec6d1d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3eeb7d57204473aa34648dd2ec6d1d9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44989 } }
I20260926 03:31:31.653240 3389 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3eeb7d57204473aa34648dd2ec6d1d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:31:31.653380 3389 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3eeb7d57204473aa34648dd2ec6d1d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:31:31.653570 3389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3eeb7d57204473aa34648dd2ec6d1d9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:31.653605 3392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:31.630515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52744 (local address 127.0.97.62:44989)
0926 03:31:31.631283 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:31.631294 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:31.631331 (+ 37us) server_negotiation.cc:407] Connection header received
0926 03:31:31.633433 (+ 2102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:31.633456 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:31.633850 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:31.634171 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:31.635974 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:31.637396 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:31.639085 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:31.639763 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:31.639895 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:31.644455 (+ 4560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:31.644466 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:31.644487 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:31.644555 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:31.648716 (+ 4161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:31.649454 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:31.649468 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:31.649479 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:31.649714 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:31.650165 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:31.650170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:31.650174 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:31.650927 (+ 753us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:31.651213 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:31.651660 (+ 447us) server_negotiation.cc:299] Negotiation successful
0926 03:31:31.652104 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":359,"threads_started":1}
I20260926 03:31:31.657096 3389 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3eeb7d57204473aa34648dd2ec6d1d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3eeb7d57204473aa34648dd2ec6d1d9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44989 } }
I20260926 03:31:31.657727 3389 leader_election.cc:302] T 00000000000000000000000000000000 P c3eeb7d57204473aa34648dd2ec6d1d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3eeb7d57204473aa34648dd2ec6d1d9; no voters:
I20260926 03:31:31.658429 3389 leader_election.cc:288] T 00000000000000000000000000000000 P c3eeb7d57204473aa34648dd2ec6d1d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:31:31.658588 3394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3eeb7d57204473aa34648dd2ec6d1d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:31:31.659761 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:31:31.659759 3394 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3eeb7d57204473aa34648dd2ec6d1d9 [term 1 LEADER]: Becoming Leader. State: Replica: c3eeb7d57204473aa34648dd2ec6d1d9, State: Running, Role: LEADER
I20260926 03:31:31.660337 3394 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3eeb7d57204473aa34648dd2ec6d1d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3eeb7d57204473aa34648dd2ec6d1d9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44989 } }
I20260926 03:31:31.660799 3389 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3eeb7d57204473aa34648dd2ec6d1d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:31:31.665645 3396 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3eeb7d57204473aa34648dd2ec6d1d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3eeb7d57204473aa34648dd2ec6d1d9. Latest consensus state: current_term: 1 leader_uuid: "c3eeb7d57204473aa34648dd2ec6d1d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3eeb7d57204473aa34648dd2ec6d1d9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44989 } } }
I20260926 03:31:31.665978 3396 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3eeb7d57204473aa34648dd2ec6d1d9 [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:31.666821 3400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:31:31.669207 3395 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3eeb7d57204473aa34648dd2ec6d1d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3eeb7d57204473aa34648dd2ec6d1d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3eeb7d57204473aa34648dd2ec6d1d9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44989 } } }
I20260926 03:31:31.669588 3395 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3eeb7d57204473aa34648dd2ec6d1d9 [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:31.681533 3400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:31:31.703451 3400 catalog_manager.cc:1347] Generated new cluster ID: 73082e745c324d7385de1f7d4d08a329
I20260926 03:31:31.703579 3400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:31:31.715561 3392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:31.706770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52758 (local address 127.0.97.62:44989)
0926 03:31:31.706938 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:31.706944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:31.707533 (+ 589us) server_negotiation.cc:407] Connection header received
0926 03:31:31.707699 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:31.707705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:31.707793 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:31.707915 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:31.708999 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:31.710167 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:31.711870 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:31.712539 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:31.712707 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:31.713862 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:31.713868 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:31.713873 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:31:31.714719 (+ 846us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:31.714796 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:31:31.714824 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:31.714949 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:31.715106 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 03:31:31.715205 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 03:31:31.717057 3400 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:31:31.718256 3400 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:31:31.729411 3400 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3eeb7d57204473aa34648dd2ec6d1d9: Generated new TSK 0
I20260926 03:31:31.730437 3400 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:31:31.744164 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 3322
I20260926 03:31:31.745303 3384 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:31:31.835515 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 3322
2026-09-26T03:31:31Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1938 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:31:31 dist-test-slave-dt7p krb5kdc[3423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:31:31 dist-test-slave-dt7p krb5kdc[3423](info): set up 2 sockets
Sep 26 03:31:31 dist-test-slave-dt7p krb5kdc[3423](info): commencing operation
krb5kdc: starting...
W20260926 03:31:32.924968 388 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.007s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:31:33 dist-test-slave-dt7p krb5kdc[3423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:31:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:31:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:31:33.118109 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34155
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34155
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:33.436337 3436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:33.436697 3436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:33.436872 3436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:33.444397 3436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:31:33.444504 3436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:33.444613 3436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:33.444692 3436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:31:33.444779 3436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:31:33.454730 3436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:33.456065 3436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:33.457423 3436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:33.468890 3442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:33.468854 3444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:33.469085 3441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:33.469851 3436 server_base.cc:1034] running on GCE node
I20260926 03:31:33.470657 3436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:33.472050 3436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:33.473271 3436 hybrid_clock.cc:648] HybridClock initialized: now 1790393493473222 us; error 36 us; skew 500 ppm
I20260926 03:31:33.473609 3436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:33 dist-test-slave-dt7p krb5kdc[3423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:33.476724 3436 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:31:33.478143 3436 webserver.cc:460] Webserver started at http://127.0.97.62:35549/ using document root <none> and password file <none>
I20260926 03:31:33.478791 3436 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:33.478927 3436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:33.479228 3436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:33.482156 3436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "daa950893deb45149c923d121ae3bfef"
format_stamp: "Formatted at 2026-09-26 03:31:33 on dist-test-slave-dt7p"
server_key: "fb7938e0a10523b3ed6c180ad9a001f7"
server_key_iv: "d5eda9bb49ff0f9a9dd7f5df6bbb3b50"
server_key_version: "encryptionkey@0"
I20260926 03:31:33.482926 3436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "daa950893deb45149c923d121ae3bfef"
format_stamp: "Formatted at 2026-09-26 03:31:33 on dist-test-slave-dt7p"
server_key: "fb7938e0a10523b3ed6c180ad9a001f7"
server_key_iv: "d5eda9bb49ff0f9a9dd7f5df6bbb3b50"
server_key_version: "encryptionkey@0"
I20260926 03:31:33.488098 3436 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:31:33.491614 3451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:33.493084 3436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:31:33.493266 3436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "daa950893deb45149c923d121ae3bfef"
format_stamp: "Formatted at 2026-09-26 03:31:33 on dist-test-slave-dt7p"
server_key: "fb7938e0a10523b3ed6c180ad9a001f7"
server_key_iv: "d5eda9bb49ff0f9a9dd7f5df6bbb3b50"
server_key_version: "encryptionkey@0"
I20260926 03:31:33.493479 3436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:33.536978 3436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:33.541419 3436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:33.541738 3436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:33.559918 3436 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34155
I20260926 03:31:33.559969 3502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34155 every 8 connection(s)
I20260926 03:31:33.561362 3436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:31:33.565037 3503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:33.566406 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 3436
I20260926 03:31:33.566664 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:31:33.567271 388 external_mini_cluster.cc:1428] Setting key d15312ca8b2f0999c7463220f38a2bdd
I20260926 03:31:33.578894 3503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daa950893deb45149c923d121ae3bfef: Bootstrap starting.
Sep 26 03:31:33 dist-test-slave-dt7p krb5kdc[3423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:31:33.581848 3503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P daa950893deb45149c923d121ae3bfef: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:33.583056 3503 log.cc:824] T 00000000000000000000000000000000 P daa950893deb45149c923d121ae3bfef: Log is configured to *not* fsync() on all Append() calls
I20260926 03:31:33.586581 3503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daa950893deb45149c923d121ae3bfef: No bootstrap required, opened a new log
I20260926 03:31:33.590828 3506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:33.569328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49808 (local address 127.0.97.62:34155)
0926 03:31:33.570141 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:33.570157 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:33.570204 (+ 47us) server_negotiation.cc:407] Connection header received
0926 03:31:33.571870 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:33.571897 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:33.572401 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:33.572810 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:33.574090 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:33.576177 (+ 2087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:33.577460 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:33.578456 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:33.578639 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:33.582438 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:33.582450 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:33.582467 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:33.582507 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:33.586408 (+ 3901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:33.587118 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:33.587128 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:33.587142 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:33.587333 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:33.587676 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:33.587680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:33.587683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:33.588414 (+ 731us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:33.588769 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:33.589074 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 03:31:33.589503 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":319,"threads_started":1}
I20260926 03:31:33.595609 3503 raft_consensus.cc:348] T 00000000000000000000000000000000 P daa950893deb45149c923d121ae3bfef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa950893deb45149c923d121ae3bfef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34155 } }
I20260926 03:31:33.595908 3503 raft_consensus.cc:374] T 00000000000000000000000000000000 P daa950893deb45149c923d121ae3bfef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:33.596020 3503 raft_consensus.cc:729] T 00000000000000000000000000000000 P daa950893deb45149c923d121ae3bfef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daa950893deb45149c923d121ae3bfef, State: Initialized, Role: FOLLOWER
I20260926 03:31:33.596675 3503 consensus_queue.cc:260] T 00000000000000000000000000000000 P daa950893deb45149c923d121ae3bfef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa950893deb45149c923d121ae3bfef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34155 } }
I20260926 03:31:33.596961 3503 raft_consensus.cc:388] T 00000000000000000000000000000000 P daa950893deb45149c923d121ae3bfef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:31:33.597131 3503 raft_consensus.cc:482] T 00000000000000000000000000000000 P daa950893deb45149c923d121ae3bfef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:31:33.597319 3503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P daa950893deb45149c923d121ae3bfef [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:33.599464 3503 raft_consensus.cc:504] T 00000000000000000000000000000000 P daa950893deb45149c923d121ae3bfef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa950893deb45149c923d121ae3bfef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34155 } }
I20260926 03:31:33.599946 3503 leader_election.cc:302] T 00000000000000000000000000000000 P daa950893deb45149c923d121ae3bfef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: daa950893deb45149c923d121ae3bfef; no voters:
I20260926 03:31:33.600549 3503 leader_election.cc:288] T 00000000000000000000000000000000 P daa950893deb45149c923d121ae3bfef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:31:33.600694 3508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P daa950893deb45149c923d121ae3bfef [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:31:33.601917 3508 raft_consensus.cc:686] T 00000000000000000000000000000000 P daa950893deb45149c923d121ae3bfef [term 1 LEADER]: Becoming Leader. State: Replica: daa950893deb45149c923d121ae3bfef, State: Running, Role: LEADER
I20260926 03:31:33.602453 3508 consensus_queue.cc:237] T 00000000000000000000000000000000 P daa950893deb45149c923d121ae3bfef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa950893deb45149c923d121ae3bfef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34155 } }
I20260926 03:31:33.602819 3503 sys_catalog.cc:564] T 00000000000000000000000000000000 P daa950893deb45149c923d121ae3bfef [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:31:33.605235 3510 sys_catalog.cc:455] T 00000000000000000000000000000000 P daa950893deb45149c923d121ae3bfef [sys.catalog]: SysCatalogTable state changed. Reason: New leader daa950893deb45149c923d121ae3bfef. Latest consensus state: current_term: 1 leader_uuid: "daa950893deb45149c923d121ae3bfef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa950893deb45149c923d121ae3bfef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34155 } } }
I20260926 03:31:33.605624 3510 sys_catalog.cc:458] T 00000000000000000000000000000000 P daa950893deb45149c923d121ae3bfef [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:33.606141 3509 sys_catalog.cc:455] T 00000000000000000000000000000000 P daa950893deb45149c923d121ae3bfef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "daa950893deb45149c923d121ae3bfef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa950893deb45149c923d121ae3bfef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34155 } } }
I20260926 03:31:33.606447 3509 sys_catalog.cc:458] T 00000000000000000000000000000000 P daa950893deb45149c923d121ae3bfef [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:33.609392 3517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:31:33.613718 3517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:31:33.622356 3517 catalog_manager.cc:1347] Generated new cluster ID: dc32192e4c9447eb8488c5e9440086b8
I20260926 03:31:33.622479 3517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:31:33.652489 3517 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:31:33.653851 3517 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:31:33.660320 3517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P daa950893deb45149c923d121ae3bfef: Generated new TSK 0
I20260926 03:31:33.661410 3517 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:31:33.708079 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34155
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:34.035161 3529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:34.035456 3529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:34.035589 3529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:34.042987 3529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:34.043119 3529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:34.043298 3529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:31:34.053697 3529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34155
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:34.055087 3529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:34.056445 3529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:34.069463 3535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:34.069523 3534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:34.069432 3537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:34.070255 3529 server_base.cc:1034] running on GCE node
I20260926 03:31:34.070955 3529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:34.071928 3529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:34.073129 3529 hybrid_clock.cc:648] HybridClock initialized: now 1790393494073082 us; error 36 us; skew 500 ppm
I20260926 03:31:34.073454 3529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:34 dist-test-slave-dt7p krb5kdc[3423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:34.076787 3529 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:31:34.078298 3529 webserver.cc:460] Webserver started at http://127.0.97.1:33973/ using document root <none> and password file <none>
I20260926 03:31:34.079023 3529 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:34.079126 3529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:34.079468 3529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:34.082583 3529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-0/data/instance:
uuid: "388217f017e9435a8af9a45113e5033d"
format_stamp: "Formatted at 2026-09-26 03:31:34 on dist-test-slave-dt7p"
server_key: "040c00306fce95ddcaa23877037ad7f9"
server_key_iv: "9572c5f8dc91bff2bb440dc195e72b4a"
server_key_version: "encryptionkey@0"
I20260926 03:31:34.083416 3529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-0/wal/instance:
uuid: "388217f017e9435a8af9a45113e5033d"
format_stamp: "Formatted at 2026-09-26 03:31:34 on dist-test-slave-dt7p"
server_key: "040c00306fce95ddcaa23877037ad7f9"
server_key_iv: "9572c5f8dc91bff2bb440dc195e72b4a"
server_key_version: "encryptionkey@0"
I20260926 03:31:34.088783 3529 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 03:31:34.092178 3544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:34.093640 3529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:31:34.093818 3529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-0/wal
uuid: "388217f017e9435a8af9a45113e5033d"
format_stamp: "Formatted at 2026-09-26 03:31:34 on dist-test-slave-dt7p"
server_key: "040c00306fce95ddcaa23877037ad7f9"
server_key_iv: "9572c5f8dc91bff2bb440dc195e72b4a"
server_key_version: "encryptionkey@0"
I20260926 03:31:34.094017 3529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:34.128835 3529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:34.132994 3529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:34.133302 3529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:34.134275 3529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:31:34.135692 3529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:31:34.135790 3529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:34.135944 3529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:31:34.136018 3529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:34.169380 3529 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38967
I20260926 03:31:34.169430 3656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38967 every 8 connection(s)
I20260926 03:31:34.170670 3529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:31:34.176617 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 3529
I20260926 03:31:34.176849 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:31:34.177489 388 external_mini_cluster.cc:1428] Setting key 2e262a1a45e4bff7e088125d2950fdd3
Sep 26 03:31:34 dist-test-slave-dt7p krb5kdc[3423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:31:34.191043 3659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:34.173219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37209 (local address 127.0.97.62:34155)
0926 03:31:34.173662 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:34.173667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:34.174454 (+ 787us) server_negotiation.cc:407] Connection header received
0926 03:31:34.176236 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:34.176241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:34.176328 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:34.176444 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:34.178109 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:34.178979 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:34.180656 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:34.181415 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:34.181575 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:34.185195 (+ 3620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:34.185205 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:34.185213 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:34.185255 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:34.187898 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:34.188738 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:34.188746 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:34.188751 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:34.188872 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:34.189307 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:34.189311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:34.189315 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:34.189580 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:34.189746 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:34.190434 (+ 688us) server_negotiation.cc:299] Negotiation successful
0926 03:31:34.190671 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":240,"threads_started":1}
I20260926 03:31:34.192056 3660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:34.173663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34155 (local address 127.0.97.1:37209)
0926 03:31:34.174310 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:34.174345 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:34.175951 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:34.176724 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:34.176733 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:34.177321 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:34.177917 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:34.177933 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:34.179152 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:34.179157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:34.180469 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:34.180480 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:34.181642 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:34.181646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:34.181853 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:34.182565 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:34.182591 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:34.184951 (+ 2360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:34.188093 (+ 3142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:34.188103 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:34.188122 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:34.188610 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:34.188989 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:34.188996 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:34.188999 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:34.189188 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:34.189707 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:34.189716 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:34.190069 (+ 353us) client_negotiation.cc:769] Sending connection context
0926 03:31:34.190337 (+ 268us) client_negotiation.cc:240] Negotiation successful
0926 03:31:34.190908 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":263,"threads_started":1}
I20260926 03:31:34.193924 3657 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34155
I20260926 03:31:34.194247 3657 heartbeater.cc:464] Registering TS with master...
I20260926 03:31:34.195001 3657 heartbeater.cc:511] Master 127.0.97.62:34155 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 03:31:34.197314 3468 ts_manager.cc:194] Registered new tserver with Master: 388217f017e9435a8af9a45113e5033d (127.0.97.1:38967)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 03:31:34.199067 3468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:37209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:31:34.223685 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34155
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:34.550352 3663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:34.550678 3663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:34.550843 3663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:34.558688 3663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:34.558836 3663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:34.559052 3663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:31:34.569294 3663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34155
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:34.570590 3663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:34.571920 3663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:34.585296 3668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:34.585369 3669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:34.585336 3671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:34.586300 3663 server_base.cc:1034] running on GCE node
I20260926 03:31:34.586911 3663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:34.587859 3663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:34.589066 3663 hybrid_clock.cc:648] HybridClock initialized: now 1790393494589008 us; error 60 us; skew 500 ppm
I20260926 03:31:34.589406 3663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:34 dist-test-slave-dt7p krb5kdc[3423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:34.592815 3663 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:31:34.594264 3663 webserver.cc:460] Webserver started at http://127.0.97.2:36303/ using document root <none> and password file <none>
I20260926 03:31:34.594978 3663 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:34.595122 3663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:34.595427 3663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:34.598603 3663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-1/data/instance:
uuid: "994d74f94c764131837965bc2a07bacf"
format_stamp: "Formatted at 2026-09-26 03:31:34 on dist-test-slave-dt7p"
server_key: "e8203bd9bfcb8b835d2843abc21b1261"
server_key_iv: "c40ad796da03c5d79ba53fda4e0de7ae"
server_key_version: "encryptionkey@0"
I20260926 03:31:34.599656 3663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-1/wal/instance:
uuid: "994d74f94c764131837965bc2a07bacf"
format_stamp: "Formatted at 2026-09-26 03:31:34 on dist-test-slave-dt7p"
server_key: "e8203bd9bfcb8b835d2843abc21b1261"
server_key_iv: "c40ad796da03c5d79ba53fda4e0de7ae"
server_key_version: "encryptionkey@0"
I20260926 03:31:34.605043 3663 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 03:31:34.608464 3678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:34.609874 3663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:31:34.610086 3663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-1/wal
uuid: "994d74f94c764131837965bc2a07bacf"
format_stamp: "Formatted at 2026-09-26 03:31:34 on dist-test-slave-dt7p"
server_key: "e8203bd9bfcb8b835d2843abc21b1261"
server_key_iv: "c40ad796da03c5d79ba53fda4e0de7ae"
server_key_version: "encryptionkey@0"
I20260926 03:31:34.610288 3663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:34.630294 3663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:34.634856 3663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:34.635181 3663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:34.636219 3663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:31:34.637655 3663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:31:34.637760 3663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:34.637915 3663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:31:34.637995 3663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:34.672905 3663 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37821
I20260926 03:31:34.672940 3790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37821 every 8 connection(s)
I20260926 03:31:34.674252 3663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:31:34.683066 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 3663
I20260926 03:31:34.683274 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:31:34.683696 388 external_mini_cluster.cc:1428] Setting key c20a11f395e1a1a977026981e831384b
Sep 26 03:31:34 dist-test-slave-dt7p krb5kdc[3423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:31:34.695688 3659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:34.676950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44339 (local address 127.0.97.62:34155)
0926 03:31:34.677162 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:34.677167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:34.678365 (+ 1198us) server_negotiation.cc:407] Connection header received
0926 03:31:34.680175 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:34.680181 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:34.680278 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:34.680389 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:34.682001 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:34.682800 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:34.684644 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:34.685129 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:34.685269 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:34.688799 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:34.688805 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:34.688808 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:34.688840 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:34.692311 (+ 3471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:34.693311 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:34.693316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:34.693318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:34.693452 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:34.693951 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:34.693954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:34.693956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:34.694211 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:34.694334 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:34.695251 (+ 917us) server_negotiation.cc:299] Negotiation successful
0926 03:31:34.695431 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 03:31:34.697413 3793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:34.677461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34155 (local address 127.0.97.2:44339)
0926 03:31:34.678195 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:34.678236 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:34.679899 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:34.680703 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:34.680712 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:34.681245 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:34.681818 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:34.681835 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:34.683001 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:34.683006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:34.684470 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:34.684482 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:34.685369 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:34.685376 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:34.685611 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:34.686311 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:34.686337 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:34.688510 (+ 2173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:34.692491 (+ 3981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:34.692504 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:34.692525 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:34.693179 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:34.693570 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:34.693578 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:34.693585 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:34.693826 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:34.694343 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:34.694354 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:34.694841 (+ 487us) client_negotiation.cc:769] Sending connection context
0926 03:31:34.695166 (+ 325us) client_negotiation.cc:240] Negotiation successful
0926 03:31:34.695779 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":273,"threads_started":1}
I20260926 03:31:34.699357 3791 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34155
I20260926 03:31:34.699709 3791 heartbeater.cc:464] Registering TS with master...
I20260926 03:31:34.700724 3791 heartbeater.cc:511] Master 127.0.97.62:34155 requested a full tablet report, sending...
I20260926 03:31:34.702502 3468 ts_manager.cc:194] Registered new tserver with Master: 994d74f94c764131837965bc2a07bacf (127.0.97.2:37821)
I20260926 03:31:34.703336 3468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:44339
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:31:34.732707 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34155
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:35.064749 3796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:35.065078 3796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:35.065264 3796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:35.072697 3796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:35.072878 3796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:35.073076 3796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:31:35.083034 3796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34155
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:35.084438 3796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:35.085858 3796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:35.099267 3801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:35.099310 3802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:35.099335 3804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:35.100301 3796 server_base.cc:1034] running on GCE node
I20260926 03:31:35.101053 3796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:35.102100 3796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:35.103370 3796 hybrid_clock.cc:648] HybridClock initialized: now 1790393495103294 us; error 63 us; skew 500 ppm
I20260926 03:31:35.103775 3796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:35 dist-test-slave-dt7p krb5kdc[3423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:35.112434 3796 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:31:35.114508 3796 webserver.cc:460] Webserver started at http://127.0.97.3:45697/ using document root <none> and password file <none>
I20260926 03:31:35.115341 3796 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:35.115589 3796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:35.115964 3796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:35.119203 3796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-2/data/instance:
uuid: "18a42e67f1ac40b48a7943b19cfda7fd"
format_stamp: "Formatted at 2026-09-26 03:31:35 on dist-test-slave-dt7p"
server_key: "1ba57963e42b8f0812f297c6dd9c0ec6"
server_key_iv: "bccd41c30cbfae0b015eb48aa7eb9857"
server_key_version: "encryptionkey@0"
I20260926 03:31:35.120064 3796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-2/wal/instance:
uuid: "18a42e67f1ac40b48a7943b19cfda7fd"
format_stamp: "Formatted at 2026-09-26 03:31:35 on dist-test-slave-dt7p"
server_key: "1ba57963e42b8f0812f297c6dd9c0ec6"
server_key_iv: "bccd41c30cbfae0b015eb48aa7eb9857"
server_key_version: "encryptionkey@0"
I20260926 03:31:35.125696 3796 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 03:31:35.129503 3811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:35.131145 3796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 03:31:35.131366 3796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-2/wal
uuid: "18a42e67f1ac40b48a7943b19cfda7fd"
format_stamp: "Formatted at 2026-09-26 03:31:35 on dist-test-slave-dt7p"
server_key: "1ba57963e42b8f0812f297c6dd9c0ec6"
server_key_iv: "bccd41c30cbfae0b015eb48aa7eb9857"
server_key_version: "encryptionkey@0"
I20260926 03:31:35.131587 3796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:35.157439 3796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:35.161773 3796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:35.162106 3796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:35.163107 3796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:31:35.164479 3796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:31:35.164605 3796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:35.164765 3796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:31:35.164842 3796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:35.198743 3796 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35967
I20260926 03:31:35.198815 3923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35967 every 8 connection(s)
I20260926 03:31:35.200055 3796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:31:35.202209 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 3796
I20260926 03:31:35.202426 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790393463219689-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:31:35.202646 3657 heartbeater.cc:503] Master 127.0.97.62:34155 was elected leader, sending a full tablet report...
I20260926 03:31:35.203038 388 external_mini_cluster.cc:1428] Setting key 318f5349ce01a52238d8bdecf7b624ec
Sep 26 03:31:35 dist-test-slave-dt7p krb5kdc[3423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:31:35.223017 3927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:35.204261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59297 (local address 127.0.97.62:34155)
0926 03:31:35.205396 (+ 1135us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:35.205402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:35.205839 (+ 437us) server_negotiation.cc:407] Connection header received
0926 03:31:35.207822 (+ 1983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:35.207827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:35.207909 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:35.208042 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:35.209659 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:35.210482 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:35.212091 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:35.212501 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:35.212661 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:35.216442 (+ 3781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:35.216448 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:35.216451 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:35.216486 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:35.219841 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:35.220658 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:35.220665 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:35.220668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:35.220802 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:35.221249 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:35.221255 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:35.221258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:35.221527 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:35.221684 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:35.222413 (+ 729us) server_negotiation.cc:299] Negotiation successful
0926 03:31:35.222626 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1001,"thread_start_us":911,"threads_started":1}
I20260926 03:31:35.224025 3926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:35.203744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34155 (local address 127.0.97.3:59297)
0926 03:31:35.205679 (+ 1935us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:35.205724 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:35.207543 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:35.208283 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:35.208291 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:35.208888 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:35.209480 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:35.209497 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:35.210652 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:35.210658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:35.211937 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:35.211949 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:35.212762 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:35.212769 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:35.213011 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:35.213783 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:35.213807 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:35.216168 (+ 2361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:35.220002 (+ 3834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:35.220013 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:35.220036 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:35.220493 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:35.220919 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:35.220925 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:35.220928 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:35.221128 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:35.221664 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:35.221672 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:35.222040 (+ 368us) client_negotiation.cc:769] Sending connection context
0926 03:31:35.222317 (+ 277us) client_negotiation.cc:240] Negotiation successful
0926 03:31:35.222894 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1671,"thread_start_us":1451,"threads_started":1}
I20260926 03:31:35.225517 3924 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34155
I20260926 03:31:35.225833 3924 heartbeater.cc:464] Registering TS with master...
I20260926 03:31:35.226584 3924 heartbeater.cc:511] Master 127.0.97.62:34155 requested a full tablet report, sending...
I20260926 03:31:35.227994 3468 ts_manager.cc:194] Registered new tserver with Master: 18a42e67f1ac40b48a7943b19cfda7fd (127.0.97.3:35967)
I20260926 03:31:35.228833 3468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59297
I20260926 03:31:35.231599 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 03:31:35 dist-test-slave-dt7p krb5kdc[3423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393495, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 03:31:35 dist-test-slave-dt7p krb5kdc[3423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393495, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:31:35.279116 3927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:35.265269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49820 (local address 127.0.97.62:34155)
0926 03:31:35.265515 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:35.265524 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:35.265719 (+ 195us) server_negotiation.cc:407] Connection header received
0926 03:31:35.265874 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:35.265883 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:35.266008 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:35.266125 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:35.267466 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:35.268223 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:35.269535 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:35.269965 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:35.270076 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:35.274050 (+ 3974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:35.274055 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:35.274059 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:35.274097 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:35.276723 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:35.277405 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:35.277409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:35.277411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:35.277549 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:35.277911 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:35.277914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:35.277916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:35.278236 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:35.278421 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:35.278722 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 03:31:35.278881 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260926 03:31:35.279798 3468 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49820
I20260926 03:31:35.282080 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 3529
Sep 26 03:31:35 dist-test-slave-dt7p krb5kdc[3423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393495, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:31:35.298697 3936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:35.282975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44948 (local address 127.0.97.1:38967)
0926 03:31:35.283603 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:35.283610 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:35.283635 (+ 25us) server_negotiation.cc:407] Connection header received
0926 03:31:35.283688 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:35.283694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:35.283906 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:35.284037 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:35.285270 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:35.286290 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:35.287547 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:35.288169 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:35.288281 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:35.292346 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:35.292360 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:35.292376 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:35.292415 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:35.295433 (+ 3018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:35.296298 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:35.296307 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:35.296309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:35.296446 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:35.296971 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:35.296976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:35.296980 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:35.297359 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:35.297695 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:35.298073 (+ 378us) server_negotiation.cc:299] Negotiation successful
0926 03:31:35.298321 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":326,"threads_started":1}
W20260926 03:31:35.299940 3652 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44948
W20260926 03:31:35.301218 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 3529: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:31:35.301383 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 3529
I20260926 03:31:35.317245 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 3663
Sep 26 03:31:35 dist-test-slave-dt7p krb5kdc[3423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393495, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:31:35.332310 3937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:35.318076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52782 (local address 127.0.97.2:37821)
0926 03:31:35.318646 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:35.318653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:35.318680 (+ 27us) server_negotiation.cc:407] Connection header received
0926 03:31:35.318748 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:35.318755 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:35.318960 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:35.319097 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:35.320259 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:35.321317 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:35.322477 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:35.322994 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:35.323088 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:35.327032 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:35.327044 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:35.327058 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:35.327093 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:35.329991 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:35.330587 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:35.330595 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:35.330597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:35.330719 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:35.331027 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:35.331031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:35.331035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:35.331354 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:35.331590 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:35.331834 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 03:31:35.332027 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":293,"threads_started":1}
W20260926 03:31:35.333606 3786 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52782
W20260926 03:31:35.334738 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 3663: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:31:35.334962 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 3663
I20260926 03:31:35.350332 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 3796
Sep 26 03:31:35 dist-test-slave-dt7p krb5kdc[3423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393495, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:31:35.366321 3938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:35.351333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52826 (local address 127.0.97.3:35967)
0926 03:31:35.351906 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:35.351913 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:35.351941 (+ 28us) server_negotiation.cc:407] Connection header received
0926 03:31:35.351999 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:35.352005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:35.352220 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:35.352383 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:35.353681 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:35.354743 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:35.356084 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:35.356681 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:35.356809 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:35.360862 (+ 4053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:35.360872 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:35.360886 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:35.360916 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:35.363465 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:35.364132 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:35.364147 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:35.364149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:35.364260 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:35.364727 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:35.364734 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:35.364737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:35.365113 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:35.365426 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:35.365698 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 03:31:35.365969 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":292,"threads_started":1}
W20260926 03:31:35.367677 3919 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52826
W20260926 03:31:35.368831 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 3796: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:31:35.368966 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 3796
I20260926 03:31:35.384990 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 3436
I20260926 03:31:35.386257 3498 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:31:35.488732 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 3436
2026-09-26T03:31:35Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3658 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:31:35 dist-test-slave-dt7p krb5kdc[3942](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:31:35 dist-test-slave-dt7p krb5kdc[3942](info): set up 2 sockets
Sep 26 03:31:35 dist-test-slave-dt7p krb5kdc[3942](info): commencing operation
krb5kdc: starting...
W20260926 03:31:36.580998 388 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.005s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:31:36 dist-test-slave-dt7p krb5kdc[3942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:31:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:31:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:31:36.773033 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33923
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33923
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:37.090826 3955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:37.091105 3955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:37.091315 3955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:37.098356 3955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:31:37.098450 3955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:37.098524 3955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:37.098611 3955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:31:37.098690 3955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:31:37.108245 3955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:37.109534 3955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:37.110818 3955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:37.122604 3960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:37.122668 3961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:37.122543 3963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:37.123512 3955 server_base.cc:1034] running on GCE node
I20260926 03:31:37.124250 3955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:37.125670 3955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:37.126871 3955 hybrid_clock.cc:648] HybridClock initialized: now 1790393497126814 us; error 56 us; skew 500 ppm
I20260926 03:31:37.127223 3955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:37 dist-test-slave-dt7p krb5kdc[3942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:37.130236 3955 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:31:37.131711 3955 webserver.cc:460] Webserver started at http://127.0.97.62:44623/ using document root <none> and password file <none>
I20260926 03:31:37.132375 3955 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:37.132517 3955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:37.132850 3955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:37.135677 3955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "ff0245025de74e609b3c1b6329443e65"
format_stamp: "Formatted at 2026-09-26 03:31:37 on dist-test-slave-dt7p"
server_key: "5c2f0bb9210c66f4443a7720133dee97"
server_key_iv: "bbac374117bec59e984b278dc7a29475"
server_key_version: "encryptionkey@0"
I20260926 03:31:37.136453 3955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "ff0245025de74e609b3c1b6329443e65"
format_stamp: "Formatted at 2026-09-26 03:31:37 on dist-test-slave-dt7p"
server_key: "5c2f0bb9210c66f4443a7720133dee97"
server_key_iv: "bbac374117bec59e984b278dc7a29475"
server_key_version: "encryptionkey@0"
I20260926 03:31:37.141450 3955 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:31:37.144879 3970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:37.146242 3955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:31:37.146437 3955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "ff0245025de74e609b3c1b6329443e65"
format_stamp: "Formatted at 2026-09-26 03:31:37 on dist-test-slave-dt7p"
server_key: "5c2f0bb9210c66f4443a7720133dee97"
server_key_iv: "bbac374117bec59e984b278dc7a29475"
server_key_version: "encryptionkey@0"
I20260926 03:31:37.146649 3955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:37.177079 3955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:37.181222 3955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:37.181516 3955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:37.199887 3955 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33923
I20260926 03:31:37.199936 4021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33923 every 8 connection(s)
I20260926 03:31:37.201284 3955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:31:37.205267 4022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:37.210860 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 3955
I20260926 03:31:37.211074 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:31:37.211954 388 external_mini_cluster.cc:1428] Setting key 760521930b264cde6e105d0a3917c4bd
I20260926 03:31:37.217434 4022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff0245025de74e609b3c1b6329443e65: Bootstrap starting.
I20260926 03:31:37.221516 4022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff0245025de74e609b3c1b6329443e65: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:37.222847 4022 log.cc:824] T 00000000000000000000000000000000 P ff0245025de74e609b3c1b6329443e65: Log is configured to *not* fsync() on all Append() calls
I20260926 03:31:37.226704 4022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff0245025de74e609b3c1b6329443e65: No bootstrap required, opened a new log
Sep 26 03:31:37 dist-test-slave-dt7p krb5kdc[3942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:31:37.238241 4025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:37.214192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48102 (local address 127.0.97.62:33923)
0926 03:31:37.215043 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:37.215054 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:37.215098 (+ 44us) server_negotiation.cc:407] Connection header received
0926 03:31:37.216840 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:37.216863 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:37.217254 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:37.217620 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:37.219592 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:37.221009 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:37.223019 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:37.223924 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:37.224189 (+ 265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:37.229000 (+ 4811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:37.229012 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:37.229032 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:37.229088 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:37.233558 (+ 4470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:37.234218 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:37.234228 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:37.234238 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:37.234471 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:37.234859 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:37.234864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:37.234867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:37.235598 (+ 731us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:37.235921 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:37.236230 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 03:31:37.236746 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":403,"threads_started":1}
I20260926 03:31:37.238709 4022 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff0245025de74e609b3c1b6329443e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0245025de74e609b3c1b6329443e65" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33923 } }
I20260926 03:31:37.239123 4022 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff0245025de74e609b3c1b6329443e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:37.239267 4022 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff0245025de74e609b3c1b6329443e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff0245025de74e609b3c1b6329443e65, State: Initialized, Role: FOLLOWER
I20260926 03:31:37.239966 4022 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff0245025de74e609b3c1b6329443e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0245025de74e609b3c1b6329443e65" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33923 } }
I20260926 03:31:37.240317 4022 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff0245025de74e609b3c1b6329443e65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:31:37.240463 4022 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff0245025de74e609b3c1b6329443e65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:31:37.240751 4022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff0245025de74e609b3c1b6329443e65 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:37.243250 4022 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff0245025de74e609b3c1b6329443e65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0245025de74e609b3c1b6329443e65" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33923 } }
I20260926 03:31:37.243783 4022 leader_election.cc:302] T 00000000000000000000000000000000 P ff0245025de74e609b3c1b6329443e65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff0245025de74e609b3c1b6329443e65; no voters:
I20260926 03:31:37.244402 4022 leader_election.cc:288] T 00000000000000000000000000000000 P ff0245025de74e609b3c1b6329443e65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:31:37.244498 4027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff0245025de74e609b3c1b6329443e65 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:31:37.245308 4027 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff0245025de74e609b3c1b6329443e65 [term 1 LEADER]: Becoming Leader. State: Replica: ff0245025de74e609b3c1b6329443e65, State: Running, Role: LEADER
I20260926 03:31:37.245823 4027 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff0245025de74e609b3c1b6329443e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0245025de74e609b3c1b6329443e65" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33923 } }
I20260926 03:31:37.246707 4022 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff0245025de74e609b3c1b6329443e65 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:31:37.248939 4029 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff0245025de74e609b3c1b6329443e65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff0245025de74e609b3c1b6329443e65. Latest consensus state: current_term: 1 leader_uuid: "ff0245025de74e609b3c1b6329443e65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0245025de74e609b3c1b6329443e65" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33923 } } }
I20260926 03:31:37.249423 4029 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff0245025de74e609b3c1b6329443e65 [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:37.249423 4028 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff0245025de74e609b3c1b6329443e65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff0245025de74e609b3c1b6329443e65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0245025de74e609b3c1b6329443e65" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33923 } } }
I20260926 03:31:37.250769 4028 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff0245025de74e609b3c1b6329443e65 [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:37.251391 4036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:31:37.256961 4036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:31:37.265429 4036 catalog_manager.cc:1347] Generated new cluster ID: bf858962c49a4108b24a18968c25ad84
I20260926 03:31:37.265537 4036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:31:37.278563 4036 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:31:37.279874 4036 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:31:37.293642 4036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff0245025de74e609b3c1b6329443e65: Generated new TSK 0
I20260926 03:31:37.294447 4036 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:31:37.345209 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33923
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:37.666836 4048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:37.667148 4048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:37.667307 4048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:37.674423 4048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:37.674577 4048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:37.674757 4048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:31:37.685571 4048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:37.686863 4048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:37.688138 4048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:37.700901 4054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:37.700838 4053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:37.701341 4056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:37.701845 4048 server_base.cc:1034] running on GCE node
I20260926 03:31:37.702379 4048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:37.703286 4048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:37.704555 4048 hybrid_clock.cc:648] HybridClock initialized: now 1790393497704506 us; error 27 us; skew 500 ppm
I20260926 03:31:37.704937 4048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:37 dist-test-slave-dt7p krb5kdc[3942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:37.708055 4048 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:31:37.709432 4048 webserver.cc:460] Webserver started at http://127.0.97.1:40063/ using document root <none> and password file <none>
I20260926 03:31:37.710112 4048 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:37.710255 4048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:37.710548 4048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:37.713429 4048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-0/data/instance:
uuid: "b4c3a8e8e91f413e966375e866c43d46"
format_stamp: "Formatted at 2026-09-26 03:31:37 on dist-test-slave-dt7p"
server_key: "dc14df358682caa1ce0461585869d73f"
server_key_iv: "5b92d587031f9bdd770f95395a5ae6b0"
server_key_version: "encryptionkey@0"
I20260926 03:31:37.714268 4048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b4c3a8e8e91f413e966375e866c43d46"
format_stamp: "Formatted at 2026-09-26 03:31:37 on dist-test-slave-dt7p"
server_key: "dc14df358682caa1ce0461585869d73f"
server_key_iv: "5b92d587031f9bdd770f95395a5ae6b0"
server_key_version: "encryptionkey@0"
I20260926 03:31:37.719332 4048 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:31:37.722688 4063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:37.723986 4048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 03:31:37.724155 4048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-0/wal
uuid: "b4c3a8e8e91f413e966375e866c43d46"
format_stamp: "Formatted at 2026-09-26 03:31:37 on dist-test-slave-dt7p"
server_key: "dc14df358682caa1ce0461585869d73f"
server_key_iv: "5b92d587031f9bdd770f95395a5ae6b0"
server_key_version: "encryptionkey@0"
I20260926 03:31:37.724351 4048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:37.742741 4048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:37.746606 4048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:37.746902 4048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:37.747828 4048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:31:37.749164 4048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:31:37.749271 4048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:37.749420 4048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:31:37.749514 4048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:37.783015 4048 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44613
I20260926 03:31:37.783068 4175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44613 every 8 connection(s)
I20260926 03:31:37.784294 4048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:31:37.785163 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 4048
I20260926 03:31:37.785362 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:31:37.786164 388 external_mini_cluster.cc:1428] Setting key f63ef51faca8e08be42e4b727243fd15
Sep 26 03:31:37 dist-test-slave-dt7p krb5kdc[3942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 03:31:37.806384 4178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:37.787233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36855 (local address 127.0.97.62:33923)
0926 03:31:37.787721 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:37.787725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:37.788887 (+ 1162us) server_negotiation.cc:407] Connection header received
0926 03:31:37.790605 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:37.790610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:37.790687 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:37.790810 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:37.792240 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:37.793361 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:37.794910 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:37.795770 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:37.795906 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:37.799599 (+ 3693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:37.799608 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:37.799617 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:37.799664 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:37.803328 (+ 3664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:37.804040 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:37.804047 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:37.804052 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:37.804188 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:37.804617 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:37.804624 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:37.804630 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:37.805044 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:37.805201 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:37.805914 (+ 713us) server_negotiation.cc:299] Negotiation successful
0926 03:31:37.806081 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":270,"threads_started":1}
I20260926 03:31:37.807413 4179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:37.787883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33923 (local address 127.0.97.1:36855)
0926 03:31:37.788742 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:37.788786 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:37.790347 (+ 1561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:37.791033 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:37.791041 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:37.791551 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:37.792087 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:37.792103 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:37.793547 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:37.793551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:37.794759 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:37.794770 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:37.796028 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:37.796034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:37.796260 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:37.796979 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:37.797006 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:37.799358 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:37.803478 (+ 4120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:37.803485 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:37.803499 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:37.803915 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:37.804300 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:37.804303 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:37.804305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:37.804478 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:37.805180 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:37.805186 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:37.805589 (+ 403us) client_negotiation.cc:769] Sending connection context
0926 03:31:37.805830 (+ 241us) client_negotiation.cc:240] Negotiation successful
0926 03:31:37.806238 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":308,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 03:31:37.809101 4176 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33923
I20260926 03:31:37.809409 4176 heartbeater.cc:464] Registering TS with master...
I20260926 03:31:37.810127 4176 heartbeater.cc:511] Master 127.0.97.62:33923 requested a full tablet report, sending...
I20260926 03:31:37.812225 3987 ts_manager.cc:194] Registered new tserver with Master: b4c3a8e8e91f413e966375e866c43d46 (127.0.97.1:44613)
I20260926 03:31:37.813689 3987 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:36855
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:31:37.832489 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33923
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:38.152024 4182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:38.152281 4182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:38.152441 4182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:38.159834 4182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:38.159950 4182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:38.160133 4182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:31:38.170475 4182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:38.171787 4182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:38.173164 4182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:38.186295 4187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:38.186342 4188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:38.186378 4190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:38.187934 4182 server_base.cc:1034] running on GCE node
I20260926 03:31:38.188537 4182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:38.189431 4182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:38.190635 4182 hybrid_clock.cc:648] HybridClock initialized: now 1790393498190579 us; error 52 us; skew 500 ppm
I20260926 03:31:38.190924 4182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:38 dist-test-slave-dt7p krb5kdc[3942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:38.193945 4182 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:31:38.195214 4182 webserver.cc:460] Webserver started at http://127.0.97.2:46845/ using document root <none> and password file <none>
I20260926 03:31:38.195827 4182 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:38.195916 4182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:38.196193 4182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:38.198962 4182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-1/data/instance:
uuid: "115a19360fe24dcb900b1e6142d60e8f"
format_stamp: "Formatted at 2026-09-26 03:31:38 on dist-test-slave-dt7p"
server_key: "0d222d06eb8d1be47f6f14f79a823668"
server_key_iv: "c6ff9f2906dcee838d09d65038a9c8f1"
server_key_version: "encryptionkey@0"
I20260926 03:31:38.199975 4182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-1/wal/instance:
uuid: "115a19360fe24dcb900b1e6142d60e8f"
format_stamp: "Formatted at 2026-09-26 03:31:38 on dist-test-slave-dt7p"
server_key: "0d222d06eb8d1be47f6f14f79a823668"
server_key_iv: "c6ff9f2906dcee838d09d65038a9c8f1"
server_key_version: "encryptionkey@0"
I20260926 03:31:38.206763 4182 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260926 03:31:38.211078 4197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:38.212749 4182 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 03:31:38.212931 4182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-1/wal
uuid: "115a19360fe24dcb900b1e6142d60e8f"
format_stamp: "Formatted at 2026-09-26 03:31:38 on dist-test-slave-dt7p"
server_key: "0d222d06eb8d1be47f6f14f79a823668"
server_key_iv: "c6ff9f2906dcee838d09d65038a9c8f1"
server_key_version: "encryptionkey@0"
I20260926 03:31:38.213155 4182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:38.238394 4182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:38.242810 4182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:38.243069 4182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:38.243970 4182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:31:38.245308 4182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:31:38.245388 4182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:38.245479 4182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:31:38.245525 4182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:38.278671 4182 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46021
I20260926 03:31:38.278774 4309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46021 every 8 connection(s)
I20260926 03:31:38.279963 4182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:31:38.280342 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 4182
I20260926 03:31:38.280553 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:31:38.281276 388 external_mini_cluster.cc:1428] Setting key 2708072cc1a731ce55453eddb0a81c42
Sep 26 03:31:38 dist-test-slave-dt7p krb5kdc[3942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 03:31:38.302333 4178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:38.282687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56251 (local address 127.0.97.62:33923)
0926 03:31:38.282854 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:38.282859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:38.284321 (+ 1462us) server_negotiation.cc:407] Connection header received
0926 03:31:38.286121 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:38.286130 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:38.286223 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:38.286337 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:38.287742 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:38.288861 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:38.290646 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:38.291490 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:38.291642 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:38.295093 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:38.295103 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:38.295112 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:38.295160 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:38.299100 (+ 3940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:38.299787 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:38.299796 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:38.299802 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:38.300026 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:38.300447 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:38.300453 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:38.300456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:38.300907 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:38.301095 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:38.301743 (+ 648us) server_negotiation.cc:299] Negotiation successful
0926 03:31:38.301960 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 03:31:38.303437 4312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:38.283330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33923 (local address 127.0.97.2:56251)
0926 03:31:38.284157 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:38.284203 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:38.285865 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:38.286522 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:38.286530 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:38.287030 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:38.287578 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:38.287593 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:38.289062 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:38.289070 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:38.290498 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:38.290509 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:38.291719 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:38.291724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:38.291925 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:38.292762 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:38.292783 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:38.294870 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:38.299243 (+ 4373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:38.299250 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:38.299262 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:38.299662 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:38.300155 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:38.300159 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:38.300161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:38.300331 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:38.301039 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:38.301044 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:38.301353 (+ 309us) client_negotiation.cc:769] Sending connection context
0926 03:31:38.301663 (+ 310us) client_negotiation.cc:240] Negotiation successful
0926 03:31:38.302169 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":351,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 03:31:38.305214 4310 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33923
I20260926 03:31:38.305606 4310 heartbeater.cc:464] Registering TS with master...
I20260926 03:31:38.306556 4310 heartbeater.cc:511] Master 127.0.97.62:33923 requested a full tablet report, sending...
I20260926 03:31:38.308272 3987 ts_manager.cc:194] Registered new tserver with Master: 115a19360fe24dcb900b1e6142d60e8f (127.0.97.2:46021)
I20260926 03:31:38.309137 3987 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:56251
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:31:38.331202 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33923
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:38.651572 4315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:38.651839 4315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:38.651927 4315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:38.659227 4315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:38.659343 4315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:38.659453 4315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:31:38.669421 4315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:38.670652 4315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:38.671927 4315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:38.685602 4321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:38.686436 4320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:38.686614 4323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:38.686954 4315 server_base.cc:1034] running on GCE node
I20260926 03:31:38.687672 4315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:38.688550 4315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:38.689829 4315 hybrid_clock.cc:648] HybridClock initialized: now 1790393498689786 us; error 33 us; skew 500 ppm
I20260926 03:31:38.690176 4315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:38 dist-test-slave-dt7p krb5kdc[3942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:38.693250 4315 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:31:38.694645 4315 webserver.cc:460] Webserver started at http://127.0.97.3:46005/ using document root <none> and password file <none>
I20260926 03:31:38.695313 4315 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:38.695441 4315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:38.695749 4315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:38.698706 4315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-2/data/instance:
uuid: "19b0efb346f6419c8e026e8565a4934e"
format_stamp: "Formatted at 2026-09-26 03:31:38 on dist-test-slave-dt7p"
server_key: "0051568ac2cd94b4d6f517dfb2321db1"
server_key_iv: "f02f0910ec086cc90bf47411375c58ac"
server_key_version: "encryptionkey@0"
I20260926 03:31:38.699460 4315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-2/wal/instance:
uuid: "19b0efb346f6419c8e026e8565a4934e"
format_stamp: "Formatted at 2026-09-26 03:31:38 on dist-test-slave-dt7p"
server_key: "0051568ac2cd94b4d6f517dfb2321db1"
server_key_iv: "f02f0910ec086cc90bf47411375c58ac"
server_key_version: "encryptionkey@0"
I20260926 03:31:38.704296 4315 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260926 03:31:38.707576 4330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:38.708920 4315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:31:38.709105 4315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-2/wal
uuid: "19b0efb346f6419c8e026e8565a4934e"
format_stamp: "Formatted at 2026-09-26 03:31:38 on dist-test-slave-dt7p"
server_key: "0051568ac2cd94b4d6f517dfb2321db1"
server_key_iv: "f02f0910ec086cc90bf47411375c58ac"
server_key_version: "encryptionkey@0"
I20260926 03:31:38.709296 4315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:38.733093 4315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:38.736907 4315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:38.737191 4315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:38.738130 4315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:31:38.739363 4315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:31:38.739465 4315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:38.739614 4315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:31:38.739687 4315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:38.772469 4315 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42355
I20260926 03:31:38.772610 4442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42355 every 8 connection(s)
I20260926 03:31:38.773810 4315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:31:38.778882 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 4315
I20260926 03:31:38.779053 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790393463219689-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:31:38.779497 388 external_mini_cluster.cc:1428] Setting key 2a7b7ca0e8e7be9efcdf3df59818379b
Sep 26 03:31:38 dist-test-slave-dt7p krb5kdc[3942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:31:38.793526 4178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:38.776381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46655 (local address 127.0.97.62:33923)
0926 03:31:38.776623 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:38.776630 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:38.777795 (+ 1165us) server_negotiation.cc:407] Connection header received
0926 03:31:38.779729 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:38.779735 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:38.779824 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:38.779951 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:38.781833 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:38.782829 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:38.784332 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:38.784792 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:38.784937 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:38.788228 (+ 3291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:38.788233 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:38.788236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:38.788273 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:38.790632 (+ 2359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:38.791298 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:38.791306 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:38.791309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:38.791436 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:38.791861 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:38.791868 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:38.791871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:38.792124 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:38.792274 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:38.792974 (+ 700us) server_negotiation.cc:299] Negotiation successful
0926 03:31:38.793177 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 03:31:38.794590 4445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:38.776941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33923 (local address 127.0.97.3:46655)
0926 03:31:38.777643 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:38.777683 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:38.779430 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:38.780243 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:38.780255 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:38.780962 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:38.781675 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:38.781693 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:38.782977 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:38.782984 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:38.784175 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:38.784186 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:38.785029 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:38.785038 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:38.785254 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:38.785930 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:38.785952 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:38.788009 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:38.790768 (+ 2759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:38.790776 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:38.790791 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:38.791179 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:38.791570 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:38.791575 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:38.791579 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:38.791753 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:38.792246 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:38.792254 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:38.792628 (+ 374us) client_negotiation.cc:769] Sending connection context
0926 03:31:38.792889 (+ 261us) client_negotiation.cc:240] Negotiation successful
0926 03:31:38.793373 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":270,"threads_started":1}
I20260926 03:31:38.795799 4443 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33923
I20260926 03:31:38.796119 4443 heartbeater.cc:464] Registering TS with master...
I20260926 03:31:38.796832 4443 heartbeater.cc:511] Master 127.0.97.62:33923 requested a full tablet report, sending...
I20260926 03:31:38.798166 3987 ts_manager.cc:194] Registered new tserver with Master: 19b0efb346f6419c8e026e8565a4934e (127.0.97.3:42355)
I20260926 03:31:38.798942 3987 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46655
I20260926 03:31:38.806425 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:31:38.817332 4176 heartbeater.cc:503] Master 127.0.97.62:33923 was elected leader, sending a full tablet report...
Sep 26 03:31:38 dist-test-slave-dt7p krb5kdc[3942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 03:31:38 dist-test-slave-dt7p krb5kdc[3942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:31:38.853504 4450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:38.838892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52042 (local address 127.0.97.1:44613)
0926 03:31:38.839496 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:38.839506 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:38.839535 (+ 29us) server_negotiation.cc:407] Connection header received
0926 03:31:38.839596 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:38.839603 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:38.839812 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:38.839941 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:38.841316 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:38.842283 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:38.843538 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:38.844140 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:38.844241 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:38.848064 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:38.848074 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:38.848089 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:38.848119 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:38.851033 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:38.851654 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:38.851661 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:38.851663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:38.851773 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:38.852099 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:38.852102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:38.852104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:38.852405 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:38.852679 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:38.852920 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 03:31:38.853110 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":303,"threads_started":1}
Sep 26 03:31:38 dist-test-slave-dt7p krb5kdc[3942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:31:38.870349 4178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:38.857508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48110 (local address 127.0.97.62:33923)
0926 03:31:38.857719 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:38.857727 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:38.857920 (+ 193us) server_negotiation.cc:407] Connection header received
0926 03:31:38.858116 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:38.858123 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:38.858189 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:38.858289 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:38.859428 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:38.860138 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:38.861398 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:38.861792 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:38.861925 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:38.865796 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:38.865801 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:38.865804 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:38.865837 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:38.868149 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:38.868794 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:38.868798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:38.868800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:38.868903 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:38.869237 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:38.869240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:38.869242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:38.869528 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:38.869694 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:38.869929 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 03:31:38.870117 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 03:31:38.870936 3987 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48110
I20260926 03:31:38.871953 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 4048
I20260926 03:31:38.884469 4450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:38.872915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52050 (local address 127.0.97.1:44613)
0926 03:31:38.873153 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:38.873159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:38.873355 (+ 196us) server_negotiation.cc:407] Connection header received
0926 03:31:38.873553 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:38.873559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:38.873654 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:38.873758 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:38.874893 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:38.875559 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:38.876829 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:38.877268 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:38.877410 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:38.879899 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:38.879906 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:38.879911 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:38.879975 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:38.882403 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:38.883006 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:38.883010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:38.883012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:38.883145 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:38.883444 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:38.883447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:38.883449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:38.883667 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:38.883839 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:38.884099 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 03:31:38.884249 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 03:31:38.885095 4171 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:31:38.966944 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 4048
I20260926 03:31:38.982787 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 4182
Sep 26 03:31:38 dist-test-slave-dt7p krb5kdc[3942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:31:38.998469 4455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:38.983582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59848 (local address 127.0.97.2:46021)
0926 03:31:38.984095 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:38.984103 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:38.984130 (+ 27us) server_negotiation.cc:407] Connection header received
0926 03:31:38.984197 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:38.984203 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:38.984407 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:38.984609 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:38.985801 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:38.986767 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:38.987993 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:38.988542 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:38.988829 (+ 287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:38.992837 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:38.992848 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:38.992863 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:38.992897 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:38.995595 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:38.996218 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:38.996229 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:38.996237 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:38.996357 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:38.996818 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:38.996824 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:38.996827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:38.997185 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:38.997520 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:38.997835 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 03:31:38.998113 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":235,"threads_started":1}
I20260926 03:31:38.999827 4305 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:31:39.082134 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 4182
I20260926 03:31:39.097602 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 4315
Sep 26 03:31:39 dist-test-slave-dt7p krb5kdc[3942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:31:39.113430 4457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:39.098658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59442 (local address 127.0.97.3:42355)
0926 03:31:39.099294 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:39.099301 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:39.099331 (+ 30us) server_negotiation.cc:407] Connection header received
0926 03:31:39.099391 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:39.099397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:39.099621 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:39.099746 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:39.101060 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:39.102104 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:39.103326 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:39.103851 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:39.103943 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:39.107967 (+ 4024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:39.107980 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:39.107995 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:39.108029 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:39.110837 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:39.111402 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:39.111408 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:39.111410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:39.111536 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:39.111818 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:39.111851 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:39.111858 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:39.112251 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:39.112547 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:39.112825 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 03:31:39.113105 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":334,"threads_started":1}
I20260926 03:31:39.114675 4438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:31:39.196734 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 4315
I20260926 03:31:39.214339 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 3955
I20260926 03:31:39.215492 4017 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:31:39.317354 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 3955
2026-09-26T03:31:39Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3828 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:31:39 dist-test-slave-dt7p krb5kdc[4462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:31:39 dist-test-slave-dt7p krb5kdc[4462](info): set up 2 sockets
Sep 26 03:31:39 dist-test-slave-dt7p krb5kdc[4462](info): commencing operation
krb5kdc: starting...
W20260926 03:31:40.433799 388 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:31:40 dist-test-slave-dt7p krb5kdc[4462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:31:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:31:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:31:40.678666 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37307
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:41.050567 4475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:41.051296 4475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:41.051538 4475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:41.060611 4475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:31:41.060804 4475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:41.060916 4475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:41.061026 4475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:31:41.061120 4475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:31:41.072088 4475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:41.073632 4475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:41.075095 4475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:41.089005 4483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:41.088968 4480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:41.089087 4481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:41.089780 4475 server_base.cc:1034] running on GCE node
I20260926 03:31:41.090768 4475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:41.092360 4475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:41.093626 4475 hybrid_clock.cc:648] HybridClock initialized: now 1790393501093578 us; error 68 us; skew 500 ppm
I20260926 03:31:41.094007 4475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:41 dist-test-slave-dt7p krb5kdc[4462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:41.105450 4475 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:31:41.107056 4475 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:41.107200 4475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:41.107492 4475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:41.110756 4475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "7aeab77619c54cdb931cce24c12068e9"
format_stamp: "Formatted at 2026-09-26 03:31:41 on dist-test-slave-dt7p"
server_key: "62164955ed7cfbc06f14cb623f1c2b95"
server_key_iv: "c2ac22a4959bbbad6876f5298817bf00"
server_key_version: "encryptionkey@0"
I20260926 03:31:41.111601 4475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "7aeab77619c54cdb931cce24c12068e9"
format_stamp: "Formatted at 2026-09-26 03:31:41 on dist-test-slave-dt7p"
server_key: "62164955ed7cfbc06f14cb623f1c2b95"
server_key_iv: "c2ac22a4959bbbad6876f5298817bf00"
server_key_version: "encryptionkey@0"
I20260926 03:31:41.117791 4475 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260926 03:31:41.121554 4489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:41.123728 4475 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 03:31:41.124029 4475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "7aeab77619c54cdb931cce24c12068e9"
format_stamp: "Formatted at 2026-09-26 03:31:41 on dist-test-slave-dt7p"
server_key: "62164955ed7cfbc06f14cb623f1c2b95"
server_key_iv: "c2ac22a4959bbbad6876f5298817bf00"
server_key_version: "encryptionkey@0"
I20260926 03:31:41.124368 4475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:41.151844 4475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:41.156832 4475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:41.157191 4475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:41.178345 4540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37307 every 8 connection(s)
I20260926 03:31:41.178365 4475 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37307
I20260926 03:31:41.179435 4475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:31:41.183570 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 4475
I20260926 03:31:41.183802 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:31:41.184813 388 external_mini_cluster.cc:1428] Setting key 483c637fc756d1ea453ee148153601bf
I20260926 03:31:41.184985 4541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:41.199476 4541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aeab77619c54cdb931cce24c12068e9: Bootstrap starting.
Sep 26 03:31:41 dist-test-slave-dt7p krb5kdc[4462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:31:41.204646 4541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7aeab77619c54cdb931cce24c12068e9: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:41.206413 4541 log.cc:824] T 00000000000000000000000000000000 P 7aeab77619c54cdb931cce24c12068e9: Log is configured to *not* fsync() on all Append() calls
I20260926 03:31:41.211736 4541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aeab77619c54cdb931cce24c12068e9: No bootstrap required, opened a new log
I20260926 03:31:41.212965 4544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:41.187408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45610 (local address 127.0.97.62:37307)
0926 03:31:41.188250 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:41.188262 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:41.188306 (+ 44us) server_negotiation.cc:407] Connection header received
0926 03:31:41.190585 (+ 2279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:41.190620 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:41.191140 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:41.191659 (+ 519us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:41.193400 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:41.194726 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:41.196152 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:41.197082 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:41.197313 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:41.202195 (+ 4882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:41.202205 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:41.202224 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:41.202281 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:41.206362 (+ 4081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:41.207694 (+ 1332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:41.207707 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:41.207721 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:41.208007 (+ 286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:41.208660 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:41.208671 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:41.208677 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:41.209715 (+ 1038us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:41.210260 (+ 545us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:41.210802 (+ 542us) server_negotiation.cc:299] Negotiation successful
0926 03:31:41.211349 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":376,"threads_started":1}
I20260926 03:31:41.220865 4541 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7aeab77619c54cdb931cce24c12068e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aeab77619c54cdb931cce24c12068e9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37307 } }
I20260926 03:31:41.221308 4541 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7aeab77619c54cdb931cce24c12068e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:41.221444 4541 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7aeab77619c54cdb931cce24c12068e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aeab77619c54cdb931cce24c12068e9, State: Initialized, Role: FOLLOWER
I20260926 03:31:41.222199 4541 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7aeab77619c54cdb931cce24c12068e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aeab77619c54cdb931cce24c12068e9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37307 } }
I20260926 03:31:41.222512 4541 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7aeab77619c54cdb931cce24c12068e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:31:41.222674 4541 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7aeab77619c54cdb931cce24c12068e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:31:41.222899 4541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7aeab77619c54cdb931cce24c12068e9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:41.225353 4541 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7aeab77619c54cdb931cce24c12068e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aeab77619c54cdb931cce24c12068e9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37307 } }
I20260926 03:31:41.225944 4541 leader_election.cc:302] T 00000000000000000000000000000000 P 7aeab77619c54cdb931cce24c12068e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7aeab77619c54cdb931cce24c12068e9; no voters:
I20260926 03:31:41.226739 4541 leader_election.cc:288] T 00000000000000000000000000000000 P 7aeab77619c54cdb931cce24c12068e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:31:41.227013 4546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7aeab77619c54cdb931cce24c12068e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:31:41.228881 4546 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7aeab77619c54cdb931cce24c12068e9 [term 1 LEADER]: Becoming Leader. State: Replica: 7aeab77619c54cdb931cce24c12068e9, State: Running, Role: LEADER
I20260926 03:31:41.229394 4541 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7aeab77619c54cdb931cce24c12068e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:31:41.229573 4546 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7aeab77619c54cdb931cce24c12068e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aeab77619c54cdb931cce24c12068e9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37307 } }
I20260926 03:31:41.232398 4548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aeab77619c54cdb931cce24c12068e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7aeab77619c54cdb931cce24c12068e9. Latest consensus state: current_term: 1 leader_uuid: "7aeab77619c54cdb931cce24c12068e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aeab77619c54cdb931cce24c12068e9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37307 } } }
I20260926 03:31:41.232816 4548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aeab77619c54cdb931cce24c12068e9 [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:41.232862 4547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aeab77619c54cdb931cce24c12068e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7aeab77619c54cdb931cce24c12068e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aeab77619c54cdb931cce24c12068e9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37307 } } }
I20260926 03:31:41.233202 4547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aeab77619c54cdb931cce24c12068e9 [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:41.243291 4554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:31:41.249598 4554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:31:41.260888 4554 catalog_manager.cc:1347] Generated new cluster ID: 63eb73c35834433f9de51c47a48f59b2
I20260926 03:31:41.261030 4554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:31:41.296433 4554 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:31:41.298499 4554 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:31:41.306951 4554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7aeab77619c54cdb931cce24c12068e9: Generated new TSK 0
I20260926 03:31:41.308152 4554 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:31:41.377374 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37307
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:41.734110 4567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:41.734378 4567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:41.734535 4567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:41.741917 4567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:41.742056 4567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:41.742239 4567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:31:41.752324 4567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:41.753654 4567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:41.754937 4567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:41.767761 4572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:41.767838 4573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:41.767800 4575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:41.768507 4567 server_base.cc:1034] running on GCE node
I20260926 03:31:41.769119 4567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:41.769954 4567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:41.771145 4567 hybrid_clock.cc:648] HybridClock initialized: now 1790393501771095 us; error 47 us; skew 500 ppm
I20260926 03:31:41.771409 4567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:41 dist-test-slave-dt7p krb5kdc[4462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:41.774534 4567 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:31:41.775579 4567 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:41.775664 4567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:41.775902 4567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:41.778730 4567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-0/data/instance:
uuid: "e3feb5ef1fa74a2a9d0fedee6c6f9cc8"
format_stamp: "Formatted at 2026-09-26 03:31:41 on dist-test-slave-dt7p"
server_key: "50585e04fa449816e06a35b658e8ce0c"
server_key_iv: "9ace4ca01b4d297f3054cd9682cfbe7c"
server_key_version: "encryptionkey@0"
I20260926 03:31:41.779487 4567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e3feb5ef1fa74a2a9d0fedee6c6f9cc8"
format_stamp: "Formatted at 2026-09-26 03:31:41 on dist-test-slave-dt7p"
server_key: "50585e04fa449816e06a35b658e8ce0c"
server_key_iv: "9ace4ca01b4d297f3054cd9682cfbe7c"
server_key_version: "encryptionkey@0"
I20260926 03:31:41.784291 4567 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260926 03:31:41.787642 4581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:41.789003 4567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260926 03:31:41.789134 4567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-0/wal
uuid: "e3feb5ef1fa74a2a9d0fedee6c6f9cc8"
format_stamp: "Formatted at 2026-09-26 03:31:41 on dist-test-slave-dt7p"
server_key: "50585e04fa449816e06a35b658e8ce0c"
server_key_iv: "9ace4ca01b4d297f3054cd9682cfbe7c"
server_key_version: "encryptionkey@0"
I20260926 03:31:41.789302 4567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:41.800928 4567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:41.805163 4567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:41.805384 4567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:41.806383 4567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:31:41.807639 4567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:31:41.807758 4567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:41.807889 4567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:31:41.807984 4567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:41.840526 4567 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37955
I20260926 03:31:41.840657 4693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37955 every 8 connection(s)
I20260926 03:31:41.841533 4567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:31:41.849794 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 4567
I20260926 03:31:41.849995 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:31:41.850800 388 external_mini_cluster.cc:1428] Setting key 7a72742ed06eb23cca401f9c72c2e426
Sep 26 03:31:41 dist-test-slave-dt7p krb5kdc[4462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:31:41.861169 4696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:41.843971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39029 (local address 127.0.97.62:37307)
0926 03:31:41.844420 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:41.844426 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:41.845454 (+ 1028us) server_negotiation.cc:407] Connection header received
0926 03:31:41.847127 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:41.847131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:41.847194 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:41.847288 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:41.848743 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:41.849440 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:41.851724 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:41.852094 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:41.852225 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:41.855589 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:41.855594 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:41.855597 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:41.855627 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:41.858186 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:41.858893 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:41.858903 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:41.858923 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:41.859057 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:41.859471 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:41.859476 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:41.859497 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:41.859766 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:41.859928 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:41.860617 (+ 689us) server_negotiation.cc:299] Negotiation successful
0926 03:31:41.860821 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":223,"threads_started":1}
I20260926 03:31:41.862102 4697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:41.844530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37307 (local address 127.0.97.1:39029)
0926 03:31:41.845285 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:41.845325 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:41.846876 (+ 1551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:41.847474 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:41.847481 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:41.847966 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:41.848530 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:41.848546 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:41.849611 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:41.849618 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:41.851553 (+ 1935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:41.851564 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:41.852317 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:41.852325 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:41.852548 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:41.853297 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:41.853318 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:41.855368 (+ 2050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:41.858317 (+ 2949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:41.858326 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:41.858342 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:41.858770 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:41.859172 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:41.859177 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:41.859180 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:41.859359 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:41.859891 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:41.859898 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:41.860221 (+ 323us) client_negotiation.cc:769] Sending connection context
0926 03:31:41.860487 (+ 266us) client_negotiation.cc:240] Negotiation successful
0926 03:31:41.861005 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":288,"threads_started":1}
I20260926 03:31:41.863586 4694 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37307
I20260926 03:31:41.863902 4694 heartbeater.cc:464] Registering TS with master...
I20260926 03:31:41.864709 4694 heartbeater.cc:511] Master 127.0.97.62:37307 requested a full tablet report, sending...
I20260926 03:31:41.867172 4506 ts_manager.cc:194] Registered new tserver with Master: e3feb5ef1fa74a2a9d0fedee6c6f9cc8 (127.0.97.1:37955)
I20260926 03:31:41.869282 4506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39029
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:31:41.895829 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37307
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:42.227538 4700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:42.227805 4700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:42.227900 4700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:42.235173 4700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:42.235280 4700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:42.235392 4700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:31:42.245285 4700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:42.246508 4700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:42.247718 4700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:42.260599 4705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:42.260633 4708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:42.260599 4706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:42.261354 4700 server_base.cc:1034] running on GCE node
I20260926 03:31:42.261902 4700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:42.262753 4700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:42.263998 4700 hybrid_clock.cc:648] HybridClock initialized: now 1790393502263959 us; error 43 us; skew 500 ppm
I20260926 03:31:42.264324 4700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:42 dist-test-slave-dt7p krb5kdc[4462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393502, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:42.267946 4700 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:31:42.269151 4700 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:42.269286 4700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:42.269604 4700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:42.272356 4700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-1/data/instance:
uuid: "3b6c3af21a7343c3a731783a30853892"
format_stamp: "Formatted at 2026-09-26 03:31:42 on dist-test-slave-dt7p"
server_key: "e805b1fa04b98cde5491860399fd872d"
server_key_iv: "5c0b37616859fe6260dd417080037123"
server_key_version: "encryptionkey@0"
I20260926 03:31:42.273290 4700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3b6c3af21a7343c3a731783a30853892"
format_stamp: "Formatted at 2026-09-26 03:31:42 on dist-test-slave-dt7p"
server_key: "e805b1fa04b98cde5491860399fd872d"
server_key_iv: "5c0b37616859fe6260dd417080037123"
server_key_version: "encryptionkey@0"
I20260926 03:31:42.278820 4700 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:31:42.282379 4714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:42.283638 4700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:31:42.283807 4700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-1/wal
uuid: "3b6c3af21a7343c3a731783a30853892"
format_stamp: "Formatted at 2026-09-26 03:31:42 on dist-test-slave-dt7p"
server_key: "e805b1fa04b98cde5491860399fd872d"
server_key_iv: "5c0b37616859fe6260dd417080037123"
server_key_version: "encryptionkey@0"
I20260926 03:31:42.284029 4700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:42.302886 4700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:42.307204 4700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:42.307497 4700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:42.308359 4700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:31:42.309634 4700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:31:42.309715 4700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:42.309791 4700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:31:42.309844 4700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:42.342129 4700 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46151
I20260926 03:31:42.342217 4826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46151 every 8 connection(s)
I20260926 03:31:42.343041 4700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:31:42.345101 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 4700
I20260926 03:31:42.345266 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:31:42.345824 388 external_mini_cluster.cc:1428] Setting key c22f9bd02e93a6f47ebbac29b3d7ad07
Sep 26 03:31:42 dist-test-slave-dt7p krb5kdc[4462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393502, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 03:31:42.362738 4696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:42.345744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55295 (local address 127.0.97.62:37307)
0926 03:31:42.345951 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:42.345957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:42.347363 (+ 1406us) server_negotiation.cc:407] Connection header received
0926 03:31:42.349245 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:42.349250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:42.349329 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:42.349446 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:42.350859 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:42.351556 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:42.353163 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:42.353598 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:42.353730 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:42.357107 (+ 3377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:42.357113 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:42.357118 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:42.357159 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:42.359828 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:42.360527 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:42.360533 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:42.360536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:42.360707 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:42.361110 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:42.361115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:42.361118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:42.361368 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:42.361527 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:42.362167 (+ 640us) server_negotiation.cc:299] Negotiation successful
0926 03:31:42.362379 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 03:31:42.363687 4829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:42.346300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37307 (local address 127.0.97.2:55295)
0926 03:31:42.347195 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:42.347250 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:42.348986 (+ 1736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:42.349661 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:42.349669 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:42.350165 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:42.350694 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:42.350710 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:42.351708 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:42.351717 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:42.353018 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:42.353030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:42.353811 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:42.353818 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:42.354034 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:42.354751 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:42.354774 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:42.356899 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:42.359970 (+ 3071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:42.359979 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:42.359995 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:42.360402 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:42.360814 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:42.360820 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:42.360824 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:42.360990 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:42.361495 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:42.361504 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:42.361831 (+ 327us) client_negotiation.cc:769] Sending connection context
0926 03:31:42.362084 (+ 253us) client_negotiation.cc:240] Negotiation successful
0926 03:31:42.362615 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":389,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 03:31:42.365226 4827 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37307
I20260926 03:31:42.365566 4827 heartbeater.cc:464] Registering TS with master...
I20260926 03:31:42.366477 4827 heartbeater.cc:511] Master 127.0.97.62:37307 requested a full tablet report, sending...
I20260926 03:31:42.368081 4506 ts_manager.cc:194] Registered new tserver with Master: 3b6c3af21a7343c3a731783a30853892 (127.0.97.2:46151)
I20260926 03:31:42.368939 4506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55295
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:31:42.388259 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37307
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:42.702426 4832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:42.702677 4832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:42.702782 4832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:42.710471 4832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:42.710577 4832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:42.710692 4832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:31:42.720129 4832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:42.721340 4832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:42.722512 4832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:42.735373 4838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:42.735342 4840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:42.735579 4837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:42.736131 4832 server_base.cc:1034] running on GCE node
I20260926 03:31:42.736822 4832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:42.737681 4832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:42.738914 4832 hybrid_clock.cc:648] HybridClock initialized: now 1790393502738875 us; error 57 us; skew 500 ppm
I20260926 03:31:42.739238 4832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:42 dist-test-slave-dt7p krb5kdc[4462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393502, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:42.742314 4832 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:31:42.743510 4832 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:42.743630 4832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:42.743943 4832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:42.746745 4832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-2/data/instance:
uuid: "2be9debcc7ad46728a3b9b26a26811ef"
format_stamp: "Formatted at 2026-09-26 03:31:42 on dist-test-slave-dt7p"
server_key: "14465798f6718a7123927e6ffe4c53f4"
server_key_iv: "18354bc79ce95a94e5f3744f5cd8324c"
server_key_version: "encryptionkey@0"
I20260926 03:31:42.747529 4832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-2/wal/instance:
uuid: "2be9debcc7ad46728a3b9b26a26811ef"
format_stamp: "Formatted at 2026-09-26 03:31:42 on dist-test-slave-dt7p"
server_key: "14465798f6718a7123927e6ffe4c53f4"
server_key_iv: "18354bc79ce95a94e5f3744f5cd8324c"
server_key_version: "encryptionkey@0"
I20260926 03:31:42.752224 4832 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260926 03:31:42.755651 4846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:42.756951 4832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260926 03:31:42.757135 4832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-2/wal
uuid: "2be9debcc7ad46728a3b9b26a26811ef"
format_stamp: "Formatted at 2026-09-26 03:31:42 on dist-test-slave-dt7p"
server_key: "14465798f6718a7123927e6ffe4c53f4"
server_key_iv: "18354bc79ce95a94e5f3744f5cd8324c"
server_key_version: "encryptionkey@0"
I20260926 03:31:42.757349 4832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:42.783728 4832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:42.788324 4832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:42.788667 4832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:42.789649 4832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:31:42.790915 4832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:31:42.791033 4832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:42.791172 4832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:31:42.791260 4832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:42.823733 4832 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38477
I20260926 03:31:42.823789 4958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38477 every 8 connection(s)
I20260926 03:31:42.824597 4832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:31:42.826262 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 4832
I20260926 03:31:42.826453 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:31:42.826937 388 external_mini_cluster.cc:1428] Setting key 3e6c7db2dc5ba05b09b85445d46679de
Sep 26 03:31:42 dist-test-slave-dt7p krb5kdc[4462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393502, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:31:42.845135 4696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:42.827281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50139 (local address 127.0.97.62:37307)
0926 03:31:42.827505 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:42.827509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:42.828965 (+ 1456us) server_negotiation.cc:407] Connection header received
0926 03:31:42.831444 (+ 2479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:42.831449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:42.831511 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:42.831599 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:42.833128 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:42.833851 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:42.835273 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:42.835655 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:42.835739 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:42.838940 (+ 3201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:42.838944 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:42.838948 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:42.838975 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:42.842068 (+ 3093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:42.842926 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:42.842930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:42.842932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:42.843049 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:42.843539 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:42.843542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:42.843544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:42.843799 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:42.843902 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:42.844734 (+ 832us) server_negotiation.cc:299] Negotiation successful
0926 03:31:42.844881 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 03:31:42.846350 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:42.827818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37307 (local address 127.0.97.3:50139)
0926 03:31:42.828782 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:42.828846 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:42.831208 (+ 2362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:42.831794 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:42.831802 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:42.832299 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:42.832979 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:42.832996 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:42.833975 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:42.833979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:42.835161 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:42.835172 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:42.835832 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:42.835836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:42.836010 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:42.836663 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:42.836686 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:42.838750 (+ 2064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:42.842179 (+ 3429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:42.842187 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:42.842201 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:42.842772 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:42.843183 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:42.843189 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:42.843194 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:42.843430 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:42.843927 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:42.843935 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:42.844314 (+ 379us) client_negotiation.cc:769] Sending connection context
0926 03:31:42.844601 (+ 287us) client_negotiation.cc:240] Negotiation successful
0926 03:31:42.845099 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":409,"threads_started":1}
I20260926 03:31:42.847558 4959 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37307
I20260926 03:31:42.847837 4959 heartbeater.cc:464] Registering TS with master...
I20260926 03:31:42.848420 4959 heartbeater.cc:511] Master 127.0.97.62:37307 requested a full tablet report, sending...
I20260926 03:31:42.849750 4506 ts_manager.cc:194] Registered new tserver with Master: 2be9debcc7ad46728a3b9b26a26811ef (127.0.97.3:38477)
I20260926 03:31:42.850517 4506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50139
I20260926 03:31:42.854291 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:31:42.869925 4696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:42.858590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34528 (local address 127.0.97.62:37307)
0926 03:31:42.858754 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:42.858762 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:42.859167 (+ 405us) server_negotiation.cc:407] Connection header received
0926 03:31:42.859350 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:42.859355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:42.859420 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:42.859516 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:42.860993 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:42.861735 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:42.862926 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:42.863320 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:42.863448 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:42.865666 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:42.865670 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:42.865673 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:42.865700 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:42.867855 (+ 2155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:42.868412 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:42.868415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:42.868418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:42.868530 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:42.868877 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:42.868880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:42.868882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:42.869151 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:42.869312 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:42.869570 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 03:31:42.869706 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 03:31:42.872162 4694 heartbeater.cc:503] Master 127.0.97.62:37307 was elected leader, sending a full tablet report...
I20260926 03:31:42.873659 4505 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34528:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:31:42.880436 4505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:31:42.912703 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:42.891442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46151 (local address 127.0.0.1:36208)
0926 03:31:42.903958 (+ 12516us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:42.904009 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:42.904222 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:42.905356 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:42.905368 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:42.905402 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:31:42.905902 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:42.905918 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:42.907821 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:42.907826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:42.909953 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:42.909966 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:42.911711 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:42.911717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:42.911886 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:42.911946 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 03:31:42.912069 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 03:31:42.912242 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12197,"mutex_wait_us":26,"thread_start_us":13168,"threads_started":1}
I20260926 03:31:42.912717 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:42.891620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36208 (local address 127.0.97.2:46151)
0926 03:31:42.902660 (+ 11040us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:42.902674 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:42.904109 (+ 1435us) server_negotiation.cc:407] Connection header received
0926 03:31:42.904522 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:42.904533 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:42.904852 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:42.905080 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:31:42.906109 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:42.907667 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:42.910124 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:42.911235 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:42.911366 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:42.911590 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:42.912232 (+ 642us) server_negotiation.cc:299] Negotiation successful
0926 03:31:42.912392 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10829,"thread_start_us":10699,"threads_started":1}
I20260926 03:31:42.913341 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:42.891828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38477 (local address 127.0.0.1:58410)
0926 03:31:42.904846 (+ 13018us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:42.904861 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:42.904985 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:42.905686 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:42.905692 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:42.905717 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:31:42.906191 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:42.906203 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:42.908065 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:42.908071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:42.910466 (+ 2395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:42.910482 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:42.912479 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:42.912486 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:42.912716 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:42.912749 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 03:31:42.912837 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 03:31:42.912951 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12900,"mutex_wait_us":1,"thread_start_us":11646,"threads_started":1}
I20260926 03:31:42.913659 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:42.892058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58410 (local address 127.0.97.3:38477)
0926 03:31:42.903075 (+ 11017us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:42.903087 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:42.905105 (+ 2018us) server_negotiation.cc:407] Connection header received
0926 03:31:42.905200 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:42.905206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:42.905410 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:42.905590 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:31:42.906417 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:42.907909 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:42.910639 (+ 2730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:42.912037 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:42.912148 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:42.912348 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:42.913212 (+ 864us) server_negotiation.cc:299] Negotiation successful
0926 03:31:42.913369 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10816,"thread_start_us":11020,"threads_started":1}
I20260926 03:31:42.914743 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:42.893624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37955 (local address 127.0.0.1:39082)
0926 03:31:42.906700 (+ 13076us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:42.906719 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:42.906839 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:42.907354 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:42.907359 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:42.907381 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:31:42.907705 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:42.907714 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:42.909694 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:42.909700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:42.911650 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:42.911664 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:42.914147 (+ 2483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:42.914153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:42.914283 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:42.914303 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 03:31:42.914364 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 03:31:42.914439 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12948,"thread_start_us":12773,"threads_started":1}
I20260926 03:31:42.915812 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:42.894386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39082 (local address 127.0.97.1:37955)
0926 03:31:42.906290 (+ 11904us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:42.906300 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:42.906823 (+ 523us) server_negotiation.cc:407] Connection header received
0926 03:31:42.906961 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:42.906968 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:42.907173 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:42.907338 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:31:42.907960 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:42.909542 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:42.911833 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:42.913701 (+ 1868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:42.913859 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:42.915272 (+ 1413us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:42.915337 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 03:31:42.915495 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11731,"thread_start_us":11560,"threads_started":1}
I20260926 03:31:42.916823 4762 tablet_service.cc:1467] Processing CreateTablet for tablet 137dc799e0234e44accc1ca23a12fa80 (DEFAULT_TABLE table=test-table [id=d5d5b2b7824d460c81f25ba920fa6ae9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:31:42.917131 4894 tablet_service.cc:1467] Processing CreateTablet for tablet 137dc799e0234e44accc1ca23a12fa80 (DEFAULT_TABLE table=test-table [id=d5d5b2b7824d460c81f25ba920fa6ae9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:31:42.918200 4894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 137dc799e0234e44accc1ca23a12fa80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:42.918229 4762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 137dc799e0234e44accc1ca23a12fa80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:42.919384 4629 tablet_service.cc:1467] Processing CreateTablet for tablet 137dc799e0234e44accc1ca23a12fa80 (DEFAULT_TABLE table=test-table [id=d5d5b2b7824d460c81f25ba920fa6ae9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:31:42.920782 4629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 137dc799e0234e44accc1ca23a12fa80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:42.929207 4977 tablet_bootstrap.cc:492] T 137dc799e0234e44accc1ca23a12fa80 P 2be9debcc7ad46728a3b9b26a26811ef: Bootstrap starting.
I20260926 03:31:42.932276 4978 tablet_bootstrap.cc:492] T 137dc799e0234e44accc1ca23a12fa80 P 3b6c3af21a7343c3a731783a30853892: Bootstrap starting.
I20260926 03:31:42.932896 4977 tablet_bootstrap.cc:654] T 137dc799e0234e44accc1ca23a12fa80 P 2be9debcc7ad46728a3b9b26a26811ef: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:42.934201 4977 log.cc:824] T 137dc799e0234e44accc1ca23a12fa80 P 2be9debcc7ad46728a3b9b26a26811ef: Log is configured to *not* fsync() on all Append() calls
I20260926 03:31:42.935686 4978 tablet_bootstrap.cc:654] T 137dc799e0234e44accc1ca23a12fa80 P 3b6c3af21a7343c3a731783a30853892: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:42.935521 4980 tablet_bootstrap.cc:492] T 137dc799e0234e44accc1ca23a12fa80 P e3feb5ef1fa74a2a9d0fedee6c6f9cc8: Bootstrap starting.
I20260926 03:31:42.936902 4978 log.cc:824] T 137dc799e0234e44accc1ca23a12fa80 P 3b6c3af21a7343c3a731783a30853892: Log is configured to *not* fsync() on all Append() calls
I20260926 03:31:42.937394 4977 tablet_bootstrap.cc:492] T 137dc799e0234e44accc1ca23a12fa80 P 2be9debcc7ad46728a3b9b26a26811ef: No bootstrap required, opened a new log
I20260926 03:31:42.937697 4977 ts_tablet_manager.cc:1397] T 137dc799e0234e44accc1ca23a12fa80 P 2be9debcc7ad46728a3b9b26a26811ef: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260926 03:31:42.939461 4980 tablet_bootstrap.cc:654] T 137dc799e0234e44accc1ca23a12fa80 P e3feb5ef1fa74a2a9d0fedee6c6f9cc8: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:42.939764 4978 tablet_bootstrap.cc:492] T 137dc799e0234e44accc1ca23a12fa80 P 3b6c3af21a7343c3a731783a30853892: No bootstrap required, opened a new log
I20260926 03:31:42.940006 4978 ts_tablet_manager.cc:1397] T 137dc799e0234e44accc1ca23a12fa80 P 3b6c3af21a7343c3a731783a30853892: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260926 03:31:42.940447 4980 log.cc:824] T 137dc799e0234e44accc1ca23a12fa80 P e3feb5ef1fa74a2a9d0fedee6c6f9cc8: Log is configured to *not* fsync() on all Append() calls
I20260926 03:31:42.943288 4980 tablet_bootstrap.cc:492] T 137dc799e0234e44accc1ca23a12fa80 P e3feb5ef1fa74a2a9d0fedee6c6f9cc8: No bootstrap required, opened a new log
I20260926 03:31:42.943529 4980 ts_tablet_manager.cc:1397] T 137dc799e0234e44accc1ca23a12fa80 P e3feb5ef1fa74a2a9d0fedee6c6f9cc8: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 03:31:42.948084 4977 raft_consensus.cc:348] T 137dc799e0234e44accc1ca23a12fa80 P 2be9debcc7ad46728a3b9b26a26811ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3af21a7343c3a731783a30853892" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46151 } } peers { permanent_uuid: "2be9debcc7ad46728a3b9b26a26811ef" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38477 } } peers { permanent_uuid: "e3feb5ef1fa74a2a9d0fedee6c6f9cc8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37955 } }
I20260926 03:31:42.948511 4977 raft_consensus.cc:374] T 137dc799e0234e44accc1ca23a12fa80 P 2be9debcc7ad46728a3b9b26a26811ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:42.948632 4977 raft_consensus.cc:729] T 137dc799e0234e44accc1ca23a12fa80 P 2be9debcc7ad46728a3b9b26a26811ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2be9debcc7ad46728a3b9b26a26811ef, State: Initialized, Role: FOLLOWER
I20260926 03:31:42.949261 4977 consensus_queue.cc:260] T 137dc799e0234e44accc1ca23a12fa80 P 2be9debcc7ad46728a3b9b26a26811ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3af21a7343c3a731783a30853892" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46151 } } peers { permanent_uuid: "2be9debcc7ad46728a3b9b26a26811ef" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38477 } } peers { permanent_uuid: "e3feb5ef1fa74a2a9d0fedee6c6f9cc8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37955 } }
I20260926 03:31:42.950294 4959 heartbeater.cc:503] Master 127.0.97.62:37307 was elected leader, sending a full tablet report...
I20260926 03:31:42.950431 4980 raft_consensus.cc:348] T 137dc799e0234e44accc1ca23a12fa80 P e3feb5ef1fa74a2a9d0fedee6c6f9cc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3af21a7343c3a731783a30853892" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46151 } } peers { permanent_uuid: "2be9debcc7ad46728a3b9b26a26811ef" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38477 } } peers { permanent_uuid: "e3feb5ef1fa74a2a9d0fedee6c6f9cc8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37955 } }
I20260926 03:31:42.950757 4980 raft_consensus.cc:374] T 137dc799e0234e44accc1ca23a12fa80 P e3feb5ef1fa74a2a9d0fedee6c6f9cc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:42.950855 4980 raft_consensus.cc:729] T 137dc799e0234e44accc1ca23a12fa80 P e3feb5ef1fa74a2a9d0fedee6c6f9cc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3feb5ef1fa74a2a9d0fedee6c6f9cc8, State: Initialized, Role: FOLLOWER
I20260926 03:31:42.950912 4977 ts_tablet_manager.cc:1428] T 137dc799e0234e44accc1ca23a12fa80 P 2be9debcc7ad46728a3b9b26a26811ef: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260926 03:31:42.950888 4978 raft_consensus.cc:348] T 137dc799e0234e44accc1ca23a12fa80 P 3b6c3af21a7343c3a731783a30853892 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3af21a7343c3a731783a30853892" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46151 } } peers { permanent_uuid: "2be9debcc7ad46728a3b9b26a26811ef" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38477 } } peers { permanent_uuid: "e3feb5ef1fa74a2a9d0fedee6c6f9cc8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37955 } }
I20260926 03:31:42.951349 4978 raft_consensus.cc:374] T 137dc799e0234e44accc1ca23a12fa80 P 3b6c3af21a7343c3a731783a30853892 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:42.951450 4978 raft_consensus.cc:729] T 137dc799e0234e44accc1ca23a12fa80 P 3b6c3af21a7343c3a731783a30853892 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6c3af21a7343c3a731783a30853892, State: Initialized, Role: FOLLOWER
I20260926 03:31:42.951418 4980 consensus_queue.cc:260] T 137dc799e0234e44accc1ca23a12fa80 P e3feb5ef1fa74a2a9d0fedee6c6f9cc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3af21a7343c3a731783a30853892" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46151 } } peers { permanent_uuid: "2be9debcc7ad46728a3b9b26a26811ef" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38477 } } peers { permanent_uuid: "e3feb5ef1fa74a2a9d0fedee6c6f9cc8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37955 } }
I20260926 03:31:42.952028 4978 consensus_queue.cc:260] T 137dc799e0234e44accc1ca23a12fa80 P 3b6c3af21a7343c3a731783a30853892 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3af21a7343c3a731783a30853892" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46151 } } peers { permanent_uuid: "2be9debcc7ad46728a3b9b26a26811ef" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38477 } } peers { permanent_uuid: "e3feb5ef1fa74a2a9d0fedee6c6f9cc8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37955 } }
I20260926 03:31:42.953826 4827 heartbeater.cc:503] Master 127.0.97.62:37307 was elected leader, sending a full tablet report...
I20260926 03:31:42.953842 4980 ts_tablet_manager.cc:1428] T 137dc799e0234e44accc1ca23a12fa80 P e3feb5ef1fa74a2a9d0fedee6c6f9cc8: Time spent starting tablet: real 0.010s user 0.009s sys 0.001s
I20260926 03:31:42.954340 4978 ts_tablet_manager.cc:1428] T 137dc799e0234e44accc1ca23a12fa80 P 3b6c3af21a7343c3a731783a30853892: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260926 03:31:42.961191 4985 raft_consensus.cc:482] T 137dc799e0234e44accc1ca23a12fa80 P 3b6c3af21a7343c3a731783a30853892 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:31:42.961418 4985 raft_consensus.cc:504] T 137dc799e0234e44accc1ca23a12fa80 P 3b6c3af21a7343c3a731783a30853892 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3af21a7343c3a731783a30853892" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46151 } } peers { permanent_uuid: "2be9debcc7ad46728a3b9b26a26811ef" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38477 } } peers { permanent_uuid: "e3feb5ef1fa74a2a9d0fedee6c6f9cc8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37955 } }
I20260926 03:31:42.962941 4985 leader_election.cc:288] T 137dc799e0234e44accc1ca23a12fa80 P 3b6c3af21a7343c3a731783a30853892 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2be9debcc7ad46728a3b9b26a26811ef (127.0.97.3:38477), e3feb5ef1fa74a2a9d0fedee6c6f9cc8 (127.0.97.1:37955)
I20260926 03:31:42.971629 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:42.963779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56891 (local address 127.0.97.1:37955)
0926 03:31:42.964131 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:42.964139 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:42.964158 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:31:42.964320 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:42.964323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:42.964381 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:42.964455 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:31:42.965236 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:42.966360 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:42.969364 (+ 3004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:42.970571 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:42.970711 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:42.970810 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:42.971262 (+ 452us) server_negotiation.cc:299] Negotiation successful
0926 03:31:42.971394 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212}
I20260926 03:31:42.971629 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:42.963233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40583 (local address 127.0.97.3:38477)
0926 03:31:42.963437 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:42.963445 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:42.963893 (+ 448us) server_negotiation.cc:407] Connection header received
0926 03:31:42.964166 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:42.964172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:42.964244 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:42.964305 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:31:42.965267 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:42.966360 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:42.969370 (+ 3010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:42.970594 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:42.970711 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:42.970810 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:42.971262 (+ 452us) server_negotiation.cc:299] Negotiation successful
0926 03:31:42.971387 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 03:31:42.971902 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:42.963126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38477 (local address 127.0.97.2:40583)
0926 03:31:42.963782 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:42.963804 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:42.964002 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:42.964497 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:42.964502 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:42.964580 (+ 78us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:31:42.965143 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:42.965156 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:42.966490 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:42.966493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:42.969199 (+ 2706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:42.969214 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:42.970839 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:42.970845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:42.971032 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:42.971134 (+ 102us) client_negotiation.cc:769] Sending connection context
0926 03:31:42.971229 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 03:31:42.971433 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":281,"spinlock_wait_cycles":6144,"thread_start_us":302,"threads_started":1}
I20260926 03:31:42.972234 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:42.963360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37955 (local address 127.0.97.2:56891)
0926 03:31:42.964103 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:42.964121 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:42.964207 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:42.964497 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:42.964502 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:42.964575 (+ 73us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:31:42.965043 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:42.965062 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:42.966487 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:42.966492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:42.969200 (+ 2708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:42.969214 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:42.970845 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:42.970850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:42.971039 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:42.971138 (+ 99us) client_negotiation.cc:769] Sending connection context
0926 03:31:42.971229 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 03:31:42.971433 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"mutex_wait_us":294,"spinlock_wait_cycles":47104,"thread_start_us":428,"threads_started":1}
I20260926 03:31:42.972913 4914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137dc799e0234e44accc1ca23a12fa80" candidate_uuid: "3b6c3af21a7343c3a731783a30853892" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2be9debcc7ad46728a3b9b26a26811ef" is_pre_election: true
I20260926 03:31:42.973073 4649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137dc799e0234e44accc1ca23a12fa80" candidate_uuid: "3b6c3af21a7343c3a731783a30853892" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3feb5ef1fa74a2a9d0fedee6c6f9cc8" is_pre_election: true
I20260926 03:31:42.973379 4914 raft_consensus.cc:2454] T 137dc799e0234e44accc1ca23a12fa80 P 2be9debcc7ad46728a3b9b26a26811ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b6c3af21a7343c3a731783a30853892 in term 0.
I20260926 03:31:42.973578 4649 raft_consensus.cc:2454] T 137dc799e0234e44accc1ca23a12fa80 P e3feb5ef1fa74a2a9d0fedee6c6f9cc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b6c3af21a7343c3a731783a30853892 in term 0.
I20260926 03:31:42.974048 4716 leader_election.cc:302] T 137dc799e0234e44accc1ca23a12fa80 P 3b6c3af21a7343c3a731783a30853892 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2be9debcc7ad46728a3b9b26a26811ef, 3b6c3af21a7343c3a731783a30853892; no voters:
I20260926 03:31:42.974413 4985 raft_consensus.cc:2781] T 137dc799e0234e44accc1ca23a12fa80 P 3b6c3af21a7343c3a731783a30853892 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:31:42.974555 4985 raft_consensus.cc:482] T 137dc799e0234e44accc1ca23a12fa80 P 3b6c3af21a7343c3a731783a30853892 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:31:42.974656 4985 raft_consensus.cc:3037] T 137dc799e0234e44accc1ca23a12fa80 P 3b6c3af21a7343c3a731783a30853892 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:42.976766 4985 raft_consensus.cc:504] T 137dc799e0234e44accc1ca23a12fa80 P 3b6c3af21a7343c3a731783a30853892 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3af21a7343c3a731783a30853892" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46151 } } peers { permanent_uuid: "2be9debcc7ad46728a3b9b26a26811ef" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38477 } } peers { permanent_uuid: "e3feb5ef1fa74a2a9d0fedee6c6f9cc8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37955 } }
I20260926 03:31:42.977352 4985 leader_election.cc:288] T 137dc799e0234e44accc1ca23a12fa80 P 3b6c3af21a7343c3a731783a30853892 [CANDIDATE]: Term 1 election: Requested vote from peers 2be9debcc7ad46728a3b9b26a26811ef (127.0.97.3:38477), e3feb5ef1fa74a2a9d0fedee6c6f9cc8 (127.0.97.1:37955)
I20260926 03:31:42.977669 4914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137dc799e0234e44accc1ca23a12fa80" candidate_uuid: "3b6c3af21a7343c3a731783a30853892" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2be9debcc7ad46728a3b9b26a26811ef"
I20260926 03:31:42.977841 4914 raft_consensus.cc:3037] T 137dc799e0234e44accc1ca23a12fa80 P 2be9debcc7ad46728a3b9b26a26811ef [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:42.977813 4649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137dc799e0234e44accc1ca23a12fa80" candidate_uuid: "3b6c3af21a7343c3a731783a30853892" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3feb5ef1fa74a2a9d0fedee6c6f9cc8"
I20260926 03:31:42.978000 4649 raft_consensus.cc:3037] T 137dc799e0234e44accc1ca23a12fa80 P e3feb5ef1fa74a2a9d0fedee6c6f9cc8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:42.979934 4914 raft_consensus.cc:2454] T 137dc799e0234e44accc1ca23a12fa80 P 2be9debcc7ad46728a3b9b26a26811ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b6c3af21a7343c3a731783a30853892 in term 1.
I20260926 03:31:42.979991 4649 raft_consensus.cc:2454] T 137dc799e0234e44accc1ca23a12fa80 P e3feb5ef1fa74a2a9d0fedee6c6f9cc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b6c3af21a7343c3a731783a30853892 in term 1.
I20260926 03:31:42.980280 4716 leader_election.cc:302] T 137dc799e0234e44accc1ca23a12fa80 P 3b6c3af21a7343c3a731783a30853892 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2be9debcc7ad46728a3b9b26a26811ef, 3b6c3af21a7343c3a731783a30853892; no voters:
I20260926 03:31:42.980605 4985 raft_consensus.cc:2781] T 137dc799e0234e44accc1ca23a12fa80 P 3b6c3af21a7343c3a731783a30853892 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:31:42.981065 4985 raft_consensus.cc:686] T 137dc799e0234e44accc1ca23a12fa80 P 3b6c3af21a7343c3a731783a30853892 [term 1 LEADER]: Becoming Leader. State: Replica: 3b6c3af21a7343c3a731783a30853892, State: Running, Role: LEADER
I20260926 03:31:42.981524 4985 consensus_queue.cc:237] T 137dc799e0234e44accc1ca23a12fa80 P 3b6c3af21a7343c3a731783a30853892 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3af21a7343c3a731783a30853892" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46151 } } peers { permanent_uuid: "2be9debcc7ad46728a3b9b26a26811ef" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38477 } } peers { permanent_uuid: "e3feb5ef1fa74a2a9d0fedee6c6f9cc8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37955 } }
I20260926 03:31:42.986018 4506 catalog_manager.cc:5495] T 137dc799e0234e44accc1ca23a12fa80 P 3b6c3af21a7343c3a731783a30853892 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b6c3af21a7343c3a731783a30853892 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "3b6c3af21a7343c3a731783a30853892" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6c3af21a7343c3a731783a30853892" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2be9debcc7ad46728a3b9b26a26811ef" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3feb5ef1fa74a2a9d0fedee6c6f9cc8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37955 } health_report { overall_health: UNKNOWN } } }
I20260926 03:31:43.013787 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:43.008713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36218 (local address 127.0.97.2:46151)
0926 03:31:43.008908 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:43.008912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:43.008930 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:31:43.009133 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:43.009137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:43.009219 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:43.009339 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:31:43.009714 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:43.010445 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:43.011745 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:43.012131 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:43.012223 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:43.012707 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:31:43.013209 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:31:43.013315 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:43.013418 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 03:31:43.013489 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 03:31:43.021624 4914 raft_consensus.cc:1261] T 137dc799e0234e44accc1ca23a12fa80 P 2be9debcc7ad46728a3b9b26a26811ef [term 1 FOLLOWER]: Refusing update from remote peer 3b6c3af21a7343c3a731783a30853892: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:31:43.021624 4649 raft_consensus.cc:1261] T 137dc799e0234e44accc1ca23a12fa80 P e3feb5ef1fa74a2a9d0fedee6c6f9cc8 [term 1 FOLLOWER]: Refusing update from remote peer 3b6c3af21a7343c3a731783a30853892: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:31:43.022775 4985 consensus_queue.cc:1035] T 137dc799e0234e44accc1ca23a12fa80 P 3b6c3af21a7343c3a731783a30853892 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be9debcc7ad46728a3b9b26a26811ef" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:31:43.023132 4988 consensus_queue.cc:1035] T 137dc799e0234e44accc1ca23a12fa80 P 3b6c3af21a7343c3a731783a30853892 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3feb5ef1fa74a2a9d0fedee6c6f9cc8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:31:43.047475 4506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:31:43.047781 4506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:31:43.051106 4506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7aeab77619c54cdb931cce24c12068e9: Sending DeleteTablet for 3 replicas of tablet 137dc799e0234e44accc1ca23a12fa80
I20260926 03:31:43.052129 4762 tablet_service.cc:1514] Processing DeleteTablet for tablet 137dc799e0234e44accc1ca23a12fa80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:31:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36208
I20260926 03:31:43.052212 4894 tablet_service.cc:1514] Processing DeleteTablet for tablet 137dc799e0234e44accc1ca23a12fa80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:31:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58410
I20260926 03:31:43.052398 4629 tablet_service.cc:1514] Processing DeleteTablet for tablet 137dc799e0234e44accc1ca23a12fa80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:31:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39082
I20260926 03:31:43.054690 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 4567
I20260926 03:31:43.064749 5001 tablet_replica.cc:330] T 137dc799e0234e44accc1ca23a12fa80 P 3b6c3af21a7343c3a731783a30853892: stopping tablet replica
I20260926 03:31:43.065196 5001 raft_consensus.cc:2229] T 137dc799e0234e44accc1ca23a12fa80 P 3b6c3af21a7343c3a731783a30853892 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:31:43.065807 5001 raft_consensus.cc:2258] T 137dc799e0234e44accc1ca23a12fa80 P 3b6c3af21a7343c3a731783a30853892 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:31:43.068395 5003 tablet_replica.cc:330] T 137dc799e0234e44accc1ca23a12fa80 P e3feb5ef1fa74a2a9d0fedee6c6f9cc8: stopping tablet replica
I20260926 03:31:43.068615 5002 tablet_replica.cc:330] T 137dc799e0234e44accc1ca23a12fa80 P 2be9debcc7ad46728a3b9b26a26811ef: stopping tablet replica
I20260926 03:31:43.069001 5002 raft_consensus.cc:2229] T 137dc799e0234e44accc1ca23a12fa80 P 2be9debcc7ad46728a3b9b26a26811ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:31:43.069468 5002 raft_consensus.cc:2258] T 137dc799e0234e44accc1ca23a12fa80 P 2be9debcc7ad46728a3b9b26a26811ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:31:43.069814 5001 ts_tablet_manager.cc:1905] T 137dc799e0234e44accc1ca23a12fa80 P 3b6c3af21a7343c3a731783a30853892: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:31:43.071455 5002 ts_tablet_manager.cc:1905] T 137dc799e0234e44accc1ca23a12fa80 P 2be9debcc7ad46728a3b9b26a26811ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:31:43.071892 5003 raft_consensus.cc:2229] T 137dc799e0234e44accc1ca23a12fa80 P e3feb5ef1fa74a2a9d0fedee6c6f9cc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:31:43.072424 5003 raft_consensus.cc:2258] T 137dc799e0234e44accc1ca23a12fa80 P e3feb5ef1fa74a2a9d0fedee6c6f9cc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:31:43.074324 5003 ts_tablet_manager.cc:1905] T 137dc799e0234e44accc1ca23a12fa80 P e3feb5ef1fa74a2a9d0fedee6c6f9cc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:31:43.077379 5001 ts_tablet_manager.cc:1918] T 137dc799e0234e44accc1ca23a12fa80 P 3b6c3af21a7343c3a731783a30853892: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:31:43.077538 5001 log.cc:1196] T 137dc799e0234e44accc1ca23a12fa80 P 3b6c3af21a7343c3a731783a30853892: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-1/wal/wals/137dc799e0234e44accc1ca23a12fa80
I20260926 03:31:43.078079 5001 ts_tablet_manager.cc:1939] T 137dc799e0234e44accc1ca23a12fa80 P 3b6c3af21a7343c3a731783a30853892: Deleting consensus metadata
I20260926 03:31:43.078773 5002 ts_tablet_manager.cc:1918] T 137dc799e0234e44accc1ca23a12fa80 P 2be9debcc7ad46728a3b9b26a26811ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:31:43.078938 5002 log.cc:1196] T 137dc799e0234e44accc1ca23a12fa80 P 2be9debcc7ad46728a3b9b26a26811ef: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-2/wal/wals/137dc799e0234e44accc1ca23a12fa80
I20260926 03:31:43.079480 5002 ts_tablet_manager.cc:1939] T 137dc799e0234e44accc1ca23a12fa80 P 2be9debcc7ad46728a3b9b26a26811ef: Deleting consensus metadata
I20260926 03:31:43.079851 4492 catalog_manager.cc:4841] TS 3b6c3af21a7343c3a731783a30853892 (127.0.97.2:46151): tablet 137dc799e0234e44accc1ca23a12fa80 (table test-table [id=d5d5b2b7824d460c81f25ba920fa6ae9]) successfully deleted
I20260926 03:31:43.081122 4491 catalog_manager.cc:4841] TS 2be9debcc7ad46728a3b9b26a26811ef (127.0.97.3:38477): tablet 137dc799e0234e44accc1ca23a12fa80 (table test-table [id=d5d5b2b7824d460c81f25ba920fa6ae9]) successfully deleted
Sep 26 03:31:43 dist-test-slave-dt7p krb5kdc[4462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:31:43.086287 5003 ts_tablet_manager.cc:1918] T 137dc799e0234e44accc1ca23a12fa80 P e3feb5ef1fa74a2a9d0fedee6c6f9cc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:31:43.086478 5003 log.cc:1196] T 137dc799e0234e44accc1ca23a12fa80 P e3feb5ef1fa74a2a9d0fedee6c6f9cc8: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790393463219689-388-0/minicluster-data/ts-0/wal/wals/137dc799e0234e44accc1ca23a12fa80
I20260926 03:31:43.087000 5003 ts_tablet_manager.cc:1939] T 137dc799e0234e44accc1ca23a12fa80 P e3feb5ef1fa74a2a9d0fedee6c6f9cc8: Deleting consensus metadata
I20260926 03:31:43.088541 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:43.056890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39088 (local address 127.0.97.1:37955)
0926 03:31:43.057066 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:43.057073 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:43.065079 (+ 8006us) server_negotiation.cc:407] Connection header received
0926 03:31:43.065501 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:43.065510 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:43.065618 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:43.065718 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:43.068634 (+ 2916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:43.069739 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:43.074517 (+ 4778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:43.075169 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:43.075268 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:43.082488 (+ 7220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:43.082499 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:43.082511 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:43.082550 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:43.085560 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:43.086235 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:43.086243 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:43.086245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:43.086436 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:43.086868 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:43.086873 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:43.086876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:43.087369 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:43.087685 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:43.087925 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 03:31:43.088131 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 03:31:43.088752 4493 catalog_manager.cc:4841] TS e3feb5ef1fa74a2a9d0fedee6c6f9cc8 (127.0.97.1:37955): tablet 137dc799e0234e44accc1ca23a12fa80 (table test-table [id=d5d5b2b7824d460c81f25ba920fa6ae9]) successfully deleted
I20260926 03:31:43.089183 4689 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:31:43.177742 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 4567
I20260926 03:31:43.201357 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 4700
Sep 26 03:31:43 dist-test-slave-dt7p krb5kdc[4462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:31:43.218286 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:43.202191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36234 (local address 127.0.97.2:46151)
0926 03:31:43.202389 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:43.202395 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:43.202418 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:31:43.202560 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:43.202566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:43.202646 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:43.202767 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:43.203883 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:43.205136 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:43.206432 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:43.207101 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:43.207238 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:43.211486 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:43.211504 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:43.211527 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:43.211589 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:43.215477 (+ 3888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:43.216136 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:43.216145 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:43.216149 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:43.216305 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:43.216699 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:43.216704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:43.216708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:43.217155 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:43.217473 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:43.217637 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 03:31:43.217857 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 03:31:43.218994 4822 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:31:43.331414 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 4700
I20260926 03:31:43.349776 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 4832
Sep 26 03:31:43 dist-test-slave-dt7p krb5kdc[4462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:31:43.365069 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:43.350500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58418 (local address 127.0.97.3:38477)
0926 03:31:43.350681 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:43.350686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:43.350719 (+ 33us) server_negotiation.cc:407] Connection header received
0926 03:31:43.350855 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:43.350860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:43.350920 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:43.350998 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:43.352234 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:43.353087 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:43.354640 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:43.355194 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:43.355303 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:43.359681 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:43.359697 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:43.359714 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:43.359753 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:43.362672 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:43.363281 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:43.363289 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:43.363291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:43.363421 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:43.363733 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:43.363736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:43.363738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:43.364070 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:43.364271 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:43.364525 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 03:31:43.364811 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 03:31:43.365697 4954 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:31:43.453926 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 4832
I20260926 03:31:43.469961 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 4475
I20260926 03:31:43.471021 4536 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:31:43.578397 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 4475
2026-09-26T03:31:43Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4262 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T03:31:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:31:43Z Disabled control of system clock
I20260926 03:31:43.644500 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45481
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 03:31:43.960742 5012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:43.960997 5012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:43.961166 5012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:43.967983 5012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:31:43.968099 5012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:43.968165 5012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:43.968264 5012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:31:43.968339 5012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:31:43.977293 5012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45481
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:43.978488 5012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:43.980222 5012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:43.991884 5020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:43.991845 5018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:43.991982 5017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:43.992854 5012 server_base.cc:1034] running on GCE node
I20260926 03:31:43.993604 5012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:43.994954 5012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:43.996188 5012 hybrid_clock.cc:648] HybridClock initialized: now 1790393503996134 us; error 44 us; skew 500 ppm
I20260926 03:31:43.996527 5012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:31:43.998749 5012 webserver.cc:460] Webserver started at http://127.0.97.62:34905/ using document root <none> and password file <none>
I20260926 03:31:43.999430 5012 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:43.999567 5012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:43.999907 5012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:44.002664 5012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "d32560a13e09469ea9a03ad892696ae2"
format_stamp: "Formatted at 2026-09-26 03:31:43 on dist-test-slave-dt7p"
server_key: "d6e33ec6d5e8caf87c34133735be12a8"
server_key_iv: "a8d818349f928da8b91c02e32b642966"
server_key_version: "encryptionkey@0"
I20260926 03:31:44.003378 5012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "d32560a13e09469ea9a03ad892696ae2"
format_stamp: "Formatted at 2026-09-26 03:31:43 on dist-test-slave-dt7p"
server_key: "d6e33ec6d5e8caf87c34133735be12a8"
server_key_iv: "a8d818349f928da8b91c02e32b642966"
server_key_version: "encryptionkey@0"
I20260926 03:31:44.008116 5012 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 03:31:44.011456 5026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:44.012802 5012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:31:44.012979 5012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "d32560a13e09469ea9a03ad892696ae2"
format_stamp: "Formatted at 2026-09-26 03:31:43 on dist-test-slave-dt7p"
server_key: "d6e33ec6d5e8caf87c34133735be12a8"
server_key_iv: "a8d818349f928da8b91c02e32b642966"
server_key_version: "encryptionkey@0"
I20260926 03:31:44.013208 5012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:44.018703 5012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:44.019443 5012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:44.019733 5012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:44.038364 5012 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45481
I20260926 03:31:44.038409 5077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45481 every 8 connection(s)
I20260926 03:31:44.039745 5012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:31:44.040887 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 5012
I20260926 03:31:44.041098 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:31:44.041837 388 external_mini_cluster.cc:1428] Setting key fcc914ecffc2e0d2561e391d1f943882
I20260926 03:31:44.044754 5078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:44.050329 5081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:44.043818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53732 (local address 127.0.97.62:45481)
0926 03:31:44.045060 (+ 1242us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:44.045076 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:44.045144 (+ 68us) server_negotiation.cc:407] Connection header received
0926 03:31:44.046766 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:44.046786 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:44.047194 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:44.047602 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:44.048695 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:44.048700 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:44.048712 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:31:44.048726 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:44.048801 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:31:44.048816 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:44.049066 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:44.049209 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 03:31:44.049449 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":414,"threads_started":1}
I20260926 03:31:44.055619 5078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d32560a13e09469ea9a03ad892696ae2: Bootstrap starting.
I20260926 03:31:44.058902 5078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d32560a13e09469ea9a03ad892696ae2: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:44.059948 5078 log.cc:824] T 00000000000000000000000000000000 P d32560a13e09469ea9a03ad892696ae2: Log is configured to *not* fsync() on all Append() calls
I20260926 03:31:44.062867 5078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d32560a13e09469ea9a03ad892696ae2: No bootstrap required, opened a new log
I20260926 03:31:44.070499 5078 raft_consensus.cc:348] T 00000000000000000000000000000000 P d32560a13e09469ea9a03ad892696ae2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d32560a13e09469ea9a03ad892696ae2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45481 } }
I20260926 03:31:44.070873 5078 raft_consensus.cc:374] T 00000000000000000000000000000000 P d32560a13e09469ea9a03ad892696ae2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:44.070971 5078 raft_consensus.cc:729] T 00000000000000000000000000000000 P d32560a13e09469ea9a03ad892696ae2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d32560a13e09469ea9a03ad892696ae2, State: Initialized, Role: FOLLOWER
I20260926 03:31:44.071600 5078 consensus_queue.cc:260] T 00000000000000000000000000000000 P d32560a13e09469ea9a03ad892696ae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d32560a13e09469ea9a03ad892696ae2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45481 } }
I20260926 03:31:44.071828 5078 raft_consensus.cc:388] T 00000000000000000000000000000000 P d32560a13e09469ea9a03ad892696ae2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:31:44.071955 5078 raft_consensus.cc:482] T 00000000000000000000000000000000 P d32560a13e09469ea9a03ad892696ae2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:31:44.072140 5078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d32560a13e09469ea9a03ad892696ae2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:44.074608 5078 raft_consensus.cc:504] T 00000000000000000000000000000000 P d32560a13e09469ea9a03ad892696ae2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d32560a13e09469ea9a03ad892696ae2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45481 } }
I20260926 03:31:44.075174 5078 leader_election.cc:302] T 00000000000000000000000000000000 P d32560a13e09469ea9a03ad892696ae2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d32560a13e09469ea9a03ad892696ae2; no voters:
I20260926 03:31:44.075846 5078 leader_election.cc:288] T 00000000000000000000000000000000 P d32560a13e09469ea9a03ad892696ae2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:31:44.075963 5083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d32560a13e09469ea9a03ad892696ae2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:31:44.077159 5083 raft_consensus.cc:686] T 00000000000000000000000000000000 P d32560a13e09469ea9a03ad892696ae2 [term 1 LEADER]: Becoming Leader. State: Replica: d32560a13e09469ea9a03ad892696ae2, State: Running, Role: LEADER
I20260926 03:31:44.077668 5083 consensus_queue.cc:237] T 00000000000000000000000000000000 P d32560a13e09469ea9a03ad892696ae2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d32560a13e09469ea9a03ad892696ae2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45481 } }
I20260926 03:31:44.077992 5078 sys_catalog.cc:564] T 00000000000000000000000000000000 P d32560a13e09469ea9a03ad892696ae2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:31:44.080775 5084 sys_catalog.cc:455] T 00000000000000000000000000000000 P d32560a13e09469ea9a03ad892696ae2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d32560a13e09469ea9a03ad892696ae2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d32560a13e09469ea9a03ad892696ae2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45481 } } }
I20260926 03:31:44.081215 5084 sys_catalog.cc:458] T 00000000000000000000000000000000 P d32560a13e09469ea9a03ad892696ae2 [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:44.080717 5085 sys_catalog.cc:455] T 00000000000000000000000000000000 P d32560a13e09469ea9a03ad892696ae2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d32560a13e09469ea9a03ad892696ae2. Latest consensus state: current_term: 1 leader_uuid: "d32560a13e09469ea9a03ad892696ae2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d32560a13e09469ea9a03ad892696ae2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45481 } } }
I20260926 03:31:44.081655 5085 sys_catalog.cc:458] T 00000000000000000000000000000000 P d32560a13e09469ea9a03ad892696ae2 [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:44.083372 5092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:31:44.087500 5092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:31:44.098502 5092 catalog_manager.cc:1347] Generated new cluster ID: 17720734eb6a4f60b51ca63620375452
I20260926 03:31:44.098626 5092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:31:44.127938 5092 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:31:44.128397 5092 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:31:44.137607 5092 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d32560a13e09469ea9a03ad892696ae2: Generated new TSK 0
I20260926 03:31:44.138397 5092 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:31:44.148202 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45481
--builtin_ntp_servers=127.0.97.20:35921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 03:31:44.474974 5102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:44.475237 5102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:44.475385 5102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:44.483464 5102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:44.483569 5102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:44.483755 5102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:31:44.493221 5102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:44.494741 5102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:44.496631 5102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:44.509501 5110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:44.509568 5107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:44.509521 5108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:44.510205 5102 server_base.cc:1034] running on GCE node
I20260926 03:31:44.510833 5102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:44.511729 5102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:44.512917 5102 hybrid_clock.cc:648] HybridClock initialized: now 1790393504512877 us; error 34 us; skew 500 ppm
I20260926 03:31:44.513234 5102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:31:44.515321 5102 webserver.cc:460] Webserver started at http://127.0.97.1:33857/ using document root <none> and password file <none>
I20260926 03:31:44.515996 5102 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:44.516147 5102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:44.516494 5102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:44.519446 5102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-0/data/instance:
uuid: "54aa4c9ccd8c4b309f3d42b5f16b351e"
format_stamp: "Formatted at 2026-09-26 03:31:44 on dist-test-slave-dt7p"
server_key: "246d0da137b1ccbdcaf86c991c6446a5"
server_key_iv: "320ed9d6fd22a5b15ebb2989870ed06e"
server_key_version: "encryptionkey@0"
I20260926 03:31:44.520180 5102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-0/wal/instance:
uuid: "54aa4c9ccd8c4b309f3d42b5f16b351e"
format_stamp: "Formatted at 2026-09-26 03:31:44 on dist-test-slave-dt7p"
server_key: "246d0da137b1ccbdcaf86c991c6446a5"
server_key_iv: "320ed9d6fd22a5b15ebb2989870ed06e"
server_key_version: "encryptionkey@0"
I20260926 03:31:44.525372 5102 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 03:31:44.528731 5116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:44.529974 5102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 03:31:44.530102 5102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-0/wal
uuid: "54aa4c9ccd8c4b309f3d42b5f16b351e"
format_stamp: "Formatted at 2026-09-26 03:31:44 on dist-test-slave-dt7p"
server_key: "246d0da137b1ccbdcaf86c991c6446a5"
server_key_iv: "320ed9d6fd22a5b15ebb2989870ed06e"
server_key_version: "encryptionkey@0"
I20260926 03:31:44.530282 5102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:44.535588 5102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:44.536284 5102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:44.536505 5102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:44.537405 5102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:31:44.538743 5102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:31:44.538825 5102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:44.538904 5102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:31:44.538959 5102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:44.572350 5102 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34219
I20260926 03:31:44.572399 5228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34219 every 8 connection(s)
I20260926 03:31:44.573710 5102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:31:44.578647 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 5102
I20260926 03:31:44.578900 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:31:44.579643 388 external_mini_cluster.cc:1428] Setting key 0e47278b1d9be697e0d246b3364e6c8f
I20260926 03:31:44.582827 5231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:44.576408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44505 (local address 127.0.97.62:45481)
0926 03:31:44.576879 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:44.576884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:44.578212 (+ 1328us) server_negotiation.cc:407] Connection header received
0926 03:31:44.580177 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:44.580183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:44.580251 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:44.580363 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:44.581923 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:44.581929 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:44.581934 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:31:44.581953 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:44.582004 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:31:44.582021 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:44.582143 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:44.582412 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 03:31:44.582481 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":247,"threads_started":1}
I20260926 03:31:44.582986 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45481
--builtin_ntp_servers=127.0.97.20:35921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 03:31:44.583745 5232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:44.577026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45481 (local address 127.0.97.1:44505)
0926 03:31:44.578059 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:44.578095 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:44.579907 (+ 1812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:44.580689 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:44.580698 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:44.580794 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:44.581651 (+ 857us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:31:44.581654 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:44.581692 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:31:44.581777 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:44.582146 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:44.582157 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:44.582164 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 03:31:44.582461 (+ 297us) client_negotiation.cc:240] Negotiation successful
0926 03:31:44.582665 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":259,"threads_started":1}
I20260926 03:31:44.585366 5229 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45481
I20260926 03:31:44.585693 5229 heartbeater.cc:464] Registering TS with master...
I20260926 03:31:44.586367 5229 heartbeater.cc:511] Master 127.0.97.62:45481 requested a full tablet report, sending...
I20260926 03:31:44.588641 5043 ts_manager.cc:194] Registered new tserver with Master: 54aa4c9ccd8c4b309f3d42b5f16b351e (127.0.97.1:34219)
W20260926 03:31:44.906865 5233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:44.907150 5233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:44.907295 5233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:44.914554 5233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:44.914651 5233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:44.914849 5233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:31:44.924093 5233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:44.925442 5233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:44.927178 5233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:44.940505 5238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:44.941135 5239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:44.941439 5241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:44.942195 5233 server_base.cc:1034] running on GCE node
I20260926 03:31:44.942907 5233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:44.943885 5233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:44.945113 5233 hybrid_clock.cc:648] HybridClock initialized: now 1790393504945049 us; error 73 us; skew 500 ppm
I20260926 03:31:44.945565 5233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:31:44.947822 5233 webserver.cc:460] Webserver started at http://127.0.97.2:34639/ using document root <none> and password file <none>
I20260926 03:31:44.948477 5233 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:44.948714 5233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:44.949023 5233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:44.951912 5233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-1/data/instance:
uuid: "0df5da17a0164499b9b20cc1824c6e90"
format_stamp: "Formatted at 2026-09-26 03:31:44 on dist-test-slave-dt7p"
server_key: "bf5f4432be1e2bee3d7d569094466103"
server_key_iv: "3616d6b4f02be8dc8ee322f9def7c0c0"
server_key_version: "encryptionkey@0"
I20260926 03:31:44.952785 5233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0df5da17a0164499b9b20cc1824c6e90"
format_stamp: "Formatted at 2026-09-26 03:31:44 on dist-test-slave-dt7p"
server_key: "bf5f4432be1e2bee3d7d569094466103"
server_key_iv: "3616d6b4f02be8dc8ee322f9def7c0c0"
server_key_version: "encryptionkey@0"
I20260926 03:31:44.957793 5233 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:31:44.961154 5247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:44.962378 5233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:31:44.962553 5233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-1/wal
uuid: "0df5da17a0164499b9b20cc1824c6e90"
format_stamp: "Formatted at 2026-09-26 03:31:44 on dist-test-slave-dt7p"
server_key: "bf5f4432be1e2bee3d7d569094466103"
server_key_iv: "3616d6b4f02be8dc8ee322f9def7c0c0"
server_key_version: "encryptionkey@0"
I20260926 03:31:44.962771 5233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:44.969040 5233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:44.969811 5233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:44.970086 5233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:44.970971 5233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:31:44.972347 5233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:31:44.972473 5233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:44.972615 5233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:31:44.972718 5233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260926 03:31:45.004879 5233 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:32989
I20260926 03:31:45.004936 5359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:32989 every 8 connection(s)
I20260926 03:31:45.006245 5233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:31:45.013638 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 5233
I20260926 03:31:45.013833 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:31:45.014320 5231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:45.008780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35249 (local address 127.0.97.62:45481)
0926 03:31:45.008950 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:45.008954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:45.010304 (+ 1350us) server_negotiation.cc:407] Connection header received
0926 03:31:45.011980 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:45.011986 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:45.012060 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:45.012167 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:45.013617 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:45.013621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:45.013625 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:31:45.013640 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:45.013675 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:31:45.013686 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:45.013844 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:45.014054 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 03:31:45.014108 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 03:31:45.014361 388 external_mini_cluster.cc:1428] Setting key 95756e18943401c417577cbabe6c4b29
I20260926 03:31:45.015580 5362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:45.009264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45481 (local address 127.0.97.2:35249)
0926 03:31:45.010150 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:45.010184 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:45.011706 (+ 1522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:45.012379 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:45.012386 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:45.012474 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:45.013246 (+ 772us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:31:45.013250 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:45.013301 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:31:45.013420 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:45.013827 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:45.013838 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:45.013849 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 03:31:45.014141 (+ 292us) client_negotiation.cc:240] Negotiation successful
0926 03:31:45.014377 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":278,"threads_started":1}
I20260926 03:31:45.017041 5360 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45481
I20260926 03:31:45.017391 5360 heartbeater.cc:464] Registering TS with master...
I20260926 03:31:45.017576 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45481
--builtin_ntp_servers=127.0.97.20:35921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 03:31:45.018100 5360 heartbeater.cc:511] Master 127.0.97.62:45481 requested a full tablet report, sending...
I20260926 03:31:45.019654 5043 ts_manager.cc:194] Registered new tserver with Master: 0df5da17a0164499b9b20cc1824c6e90 (127.0.97.2:32989)
W20260926 03:31:45.346812 5363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:45.347126 5363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:45.347251 5363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:45.354588 5363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:45.354687 5363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:45.354851 5363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:31:45.363765 5363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:45.365097 5363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:45.366915 5363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:45.380309 5368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:45.380331 5369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:45.380466 5371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:45.386562 5363 server_base.cc:1034] running on GCE node
I20260926 03:31:45.387256 5363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:45.388298 5363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:45.389521 5363 hybrid_clock.cc:648] HybridClock initialized: now 1790393505389426 us; error 81 us; skew 500 ppm
I20260926 03:31:45.389870 5363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:31:45.392278 5363 webserver.cc:460] Webserver started at http://127.0.97.3:36537/ using document root <none> and password file <none>
I20260926 03:31:45.393097 5363 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:45.393193 5363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:45.393502 5363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:45.396445 5363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-2/data/instance:
uuid: "e12f413a8222496194fe5ca8b8ddc040"
format_stamp: "Formatted at 2026-09-26 03:31:45 on dist-test-slave-dt7p"
server_key: "2dcbe1b4a11c958601ae0062ba2c1af6"
server_key_iv: "a3436aa717175d3d4912fd8107d9a1ae"
server_key_version: "encryptionkey@0"
I20260926 03:31:45.397357 5363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e12f413a8222496194fe5ca8b8ddc040"
format_stamp: "Formatted at 2026-09-26 03:31:45 on dist-test-slave-dt7p"
server_key: "2dcbe1b4a11c958601ae0062ba2c1af6"
server_key_iv: "a3436aa717175d3d4912fd8107d9a1ae"
server_key_version: "encryptionkey@0"
I20260926 03:31:45.403369 5363 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 03:31:45.407006 5377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:45.408511 5363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:31:45.408772 5363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-2/wal
uuid: "e12f413a8222496194fe5ca8b8ddc040"
format_stamp: "Formatted at 2026-09-26 03:31:45 on dist-test-slave-dt7p"
server_key: "2dcbe1b4a11c958601ae0062ba2c1af6"
server_key_iv: "a3436aa717175d3d4912fd8107d9a1ae"
server_key_version: "encryptionkey@0"
I20260926 03:31:45.409009 5363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:45.414618 5363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:45.415385 5363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:45.415658 5363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:45.416707 5363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:31:45.418109 5363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:31:45.418231 5363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:45.418367 5363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:31:45.418466 5363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:45.454888 5363 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37719
I20260926 03:31:45.454988 5489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37719 every 8 connection(s)
I20260926 03:31:45.456243 5363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:31:45.458523 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 5363
I20260926 03:31:45.458748 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:31:45.459293 388 external_mini_cluster.cc:1428] Setting key 07e1cb9e8b36bfac2b842a48900630dc
I20260926 03:31:45.466326 5231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:45.459824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35863 (local address 127.0.97.62:45481)
0926 03:31:45.460024 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:45.460029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:45.462016 (+ 1987us) server_negotiation.cc:407] Connection header received
0926 03:31:45.464029 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:45.464033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:45.464091 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:45.464182 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:45.465665 (+ 1483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:45.465670 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:45.465673 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:31:45.465689 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:45.465755 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:31:45.465767 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:45.465880 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:45.466061 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 03:31:45.466112 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 03:31:45.467504 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:45.460387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45481 (local address 127.0.97.3:35863)
0926 03:31:45.461808 (+ 1421us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:45.461863 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:45.463757 (+ 1894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:45.464431 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:45.464440 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:45.464543 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:45.465359 (+ 816us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:31:45.465361 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:45.465411 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:31:45.465498 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:45.465857 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:45.465865 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:45.465871 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 03:31:45.466108 (+ 237us) client_negotiation.cc:240] Negotiation successful
0926 03:31:45.466334 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":405,"threads_started":1}
I20260926 03:31:45.468952 5490 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45481
I20260926 03:31:45.469237 5490 heartbeater.cc:464] Registering TS with master...
I20260926 03:31:45.469859 5490 heartbeater.cc:511] Master 127.0.97.62:45481 requested a full tablet report, sending...
I20260926 03:31:45.471294 5043 ts_manager.cc:194] Registered new tserver with Master: e12f413a8222496194fe5ca8b8ddc040 (127.0.97.3:37719)
I20260926 03:31:45.475100 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:31:45.482667 5231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:45.479712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53740 (local address 127.0.97.62:45481)
0926 03:31:45.479978 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:45.479987 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:45.480645 (+ 658us) server_negotiation.cc:407] Connection header received
0926 03:31:45.480787 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:45.480791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:45.480884 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:45.480971 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:45.481959 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:45.481963 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:45.481967 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:31:45.481981 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:45.482009 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:31:45.482038 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:45.482141 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:45.482308 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 03:31:45.482377 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 03:31:45.486861 5043 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53740:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:31:45.494119 5043 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:31:45.515348 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:45.507677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37719 (local address 127.0.0.1:50802)
0926 03:31:45.509679 (+ 2002us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:45.509741 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:45.509920 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:45.512232 (+ 2312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:45.512239 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:45.512368 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:45.513541 (+ 1173us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:31:45.513547 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:45.513592 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:31:45.513697 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:45.514359 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:45.514373 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:45.514385 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 03:31:45.514597 (+ 212us) client_negotiation.cc:240] Negotiation successful
0926 03:31:45.514704 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1670,"thread_start_us":434,"threads_started":1}
I20260926 03:31:45.516258 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:45.507943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50802 (local address 127.0.97.3:37719)
0926 03:31:45.508851 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:45.508862 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:45.510151 (+ 1289us) server_negotiation.cc:407] Connection header received
0926 03:31:45.510261 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:45.510274 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:45.510616 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:45.510792 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:45.513924 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:45.513938 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:45.513971 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:31:45.514006 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:45.514078 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:31:45.514104 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:45.514296 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:45.515598 (+ 1302us) server_negotiation.cc:299] Negotiation successful
0926 03:31:45.515720 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":520,"threads_started":1}
I20260926 03:31:45.520543 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:45.508896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34219 (local address 127.0.0.1:53010)
0926 03:31:45.516632 (+ 7736us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:45.516657 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:45.516761 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:45.518543 (+ 1782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:45.518551 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:45.518611 (+ 60us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:45.519286 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:31:45.519291 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:45.519313 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:31:45.519368 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:45.519953 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:45.519961 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:45.519969 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 03:31:45.520128 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 03:31:45.520201 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7574,"threads_started":1}
I20260926 03:31:45.521412 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:45.508546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53010 (local address 127.0.97.1:34219)
0926 03:31:45.513319 (+ 4773us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:45.513328 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:45.516892 (+ 3564us) server_negotiation.cc:407] Connection header received
0926 03:31:45.516996 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:45.517007 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:45.517266 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:45.517432 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:45.519571 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:45.519583 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:45.519602 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:31:45.519636 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:45.519705 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:31:45.519729 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:45.519885 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:45.520782 (+ 897us) server_negotiation.cc:299] Negotiation successful
0926 03:31:45.520904 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4552,"thread_start_us":431,"threads_started":1}
I20260926 03:31:45.525496 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:45.509138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32989 (local address 127.0.0.1:46920)
0926 03:31:45.521918 (+ 12780us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:45.521940 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:45.522047 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:45.523218 (+ 1171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:45.523223 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:45.523248 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:45.524460 (+ 1212us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:31:45.524463 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:45.524478 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:31:45.524519 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:45.525060 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:45.525065 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:45.525070 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 03:31:45.525168 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 03:31:45.525216 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12659,"thread_start_us":13640,"threads_started":1}
I20260926 03:31:45.525557 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:45.509626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46920 (local address 127.0.97.2:32989)
0926 03:31:45.522670 (+ 13044us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:45.522694 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:45.522731 (+ 37us) server_negotiation.cc:407] Connection header received
0926 03:31:45.522814 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:45.522822 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:45.523031 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:45.523540 (+ 509us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:45.524739 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:45.524746 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:45.524759 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:31:45.524779 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:45.524834 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:31:45.524855 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:45.525116 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:45.525180 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 03:31:45.525266 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12809,"thread_start_us":12624,"threads_started":1}
I20260926 03:31:45.528159 5425 tablet_service.cc:1467] Processing CreateTablet for tablet de911a53b444427083ec1a1126da9834 (DEFAULT_TABLE table=test-table [id=3599f079c9c840bca81cbac2ca2ff0ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:31:45.528159 5164 tablet_service.cc:1467] Processing CreateTablet for tablet de911a53b444427083ec1a1126da9834 (DEFAULT_TABLE table=test-table [id=3599f079c9c840bca81cbac2ca2ff0ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:31:45.529937 5164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de911a53b444427083ec1a1126da9834. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:45.529937 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de911a53b444427083ec1a1126da9834. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:45.531669 5295 tablet_service.cc:1467] Processing CreateTablet for tablet de911a53b444427083ec1a1126da9834 (DEFAULT_TABLE table=test-table [id=3599f079c9c840bca81cbac2ca2ff0ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:31:45.533133 5295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de911a53b444427083ec1a1126da9834. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:45.546716 5510 tablet_bootstrap.cc:492] T de911a53b444427083ec1a1126da9834 P 54aa4c9ccd8c4b309f3d42b5f16b351e: Bootstrap starting.
I20260926 03:31:45.549044 5509 tablet_bootstrap.cc:492] T de911a53b444427083ec1a1126da9834 P e12f413a8222496194fe5ca8b8ddc040: Bootstrap starting.
I20260926 03:31:45.549194 5508 tablet_bootstrap.cc:492] T de911a53b444427083ec1a1126da9834 P 0df5da17a0164499b9b20cc1824c6e90: Bootstrap starting.
I20260926 03:31:45.550899 5510 tablet_bootstrap.cc:654] T de911a53b444427083ec1a1126da9834 P 54aa4c9ccd8c4b309f3d42b5f16b351e: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:45.552091 5510 log.cc:824] T de911a53b444427083ec1a1126da9834 P 54aa4c9ccd8c4b309f3d42b5f16b351e: Log is configured to *not* fsync() on all Append() calls
I20260926 03:31:45.553445 5508 tablet_bootstrap.cc:654] T de911a53b444427083ec1a1126da9834 P 0df5da17a0164499b9b20cc1824c6e90: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:45.553633 5509 tablet_bootstrap.cc:654] T de911a53b444427083ec1a1126da9834 P e12f413a8222496194fe5ca8b8ddc040: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:45.554862 5508 log.cc:824] T de911a53b444427083ec1a1126da9834 P 0df5da17a0164499b9b20cc1824c6e90: Log is configured to *not* fsync() on all Append() calls
I20260926 03:31:45.554860 5509 log.cc:824] T de911a53b444427083ec1a1126da9834 P e12f413a8222496194fe5ca8b8ddc040: Log is configured to *not* fsync() on all Append() calls
I20260926 03:31:45.555298 5510 tablet_bootstrap.cc:492] T de911a53b444427083ec1a1126da9834 P 54aa4c9ccd8c4b309f3d42b5f16b351e: No bootstrap required, opened a new log
I20260926 03:31:45.555605 5510 ts_tablet_manager.cc:1397] T de911a53b444427083ec1a1126da9834 P 54aa4c9ccd8c4b309f3d42b5f16b351e: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 03:31:45.558413 5509 tablet_bootstrap.cc:492] T de911a53b444427083ec1a1126da9834 P e12f413a8222496194fe5ca8b8ddc040: No bootstrap required, opened a new log
I20260926 03:31:45.558410 5508 tablet_bootstrap.cc:492] T de911a53b444427083ec1a1126da9834 P 0df5da17a0164499b9b20cc1824c6e90: No bootstrap required, opened a new log
I20260926 03:31:45.558699 5509 ts_tablet_manager.cc:1397] T de911a53b444427083ec1a1126da9834 P e12f413a8222496194fe5ca8b8ddc040: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260926 03:31:45.558699 5508 ts_tablet_manager.cc:1397] T de911a53b444427083ec1a1126da9834 P 0df5da17a0164499b9b20cc1824c6e90: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 03:31:45.563567 5510 raft_consensus.cc:348] T de911a53b444427083ec1a1126da9834 P 54aa4c9ccd8c4b309f3d42b5f16b351e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12f413a8222496194fe5ca8b8ddc040" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37719 } } peers { permanent_uuid: "0df5da17a0164499b9b20cc1824c6e90" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32989 } } peers { permanent_uuid: "54aa4c9ccd8c4b309f3d42b5f16b351e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34219 } }
I20260926 03:31:45.563956 5510 raft_consensus.cc:374] T de911a53b444427083ec1a1126da9834 P 54aa4c9ccd8c4b309f3d42b5f16b351e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:45.564080 5510 raft_consensus.cc:729] T de911a53b444427083ec1a1126da9834 P 54aa4c9ccd8c4b309f3d42b5f16b351e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54aa4c9ccd8c4b309f3d42b5f16b351e, State: Initialized, Role: FOLLOWER
I20260926 03:31:45.564806 5510 consensus_queue.cc:260] T de911a53b444427083ec1a1126da9834 P 54aa4c9ccd8c4b309f3d42b5f16b351e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12f413a8222496194fe5ca8b8ddc040" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37719 } } peers { permanent_uuid: "0df5da17a0164499b9b20cc1824c6e90" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32989 } } peers { permanent_uuid: "54aa4c9ccd8c4b309f3d42b5f16b351e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34219 } }
I20260926 03:31:45.565930 5229 heartbeater.cc:503] Master 127.0.97.62:45481 was elected leader, sending a full tablet report...
I20260926 03:31:45.566596 5510 ts_tablet_manager.cc:1428] T de911a53b444427083ec1a1126da9834 P 54aa4c9ccd8c4b309f3d42b5f16b351e: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20260926 03:31:45.568763 5514 raft_consensus.cc:482] T de911a53b444427083ec1a1126da9834 P 54aa4c9ccd8c4b309f3d42b5f16b351e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:31:45.569087 5514 raft_consensus.cc:504] T de911a53b444427083ec1a1126da9834 P 54aa4c9ccd8c4b309f3d42b5f16b351e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12f413a8222496194fe5ca8b8ddc040" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37719 } } peers { permanent_uuid: "0df5da17a0164499b9b20cc1824c6e90" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32989 } } peers { permanent_uuid: "54aa4c9ccd8c4b309f3d42b5f16b351e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34219 } }
I20260926 03:31:45.569958 5509 raft_consensus.cc:348] T de911a53b444427083ec1a1126da9834 P e12f413a8222496194fe5ca8b8ddc040 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12f413a8222496194fe5ca8b8ddc040" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37719 } } peers { permanent_uuid: "0df5da17a0164499b9b20cc1824c6e90" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32989 } } peers { permanent_uuid: "54aa4c9ccd8c4b309f3d42b5f16b351e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34219 } }
I20260926 03:31:45.569958 5508 raft_consensus.cc:348] T de911a53b444427083ec1a1126da9834 P 0df5da17a0164499b9b20cc1824c6e90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12f413a8222496194fe5ca8b8ddc040" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37719 } } peers { permanent_uuid: "0df5da17a0164499b9b20cc1824c6e90" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32989 } } peers { permanent_uuid: "54aa4c9ccd8c4b309f3d42b5f16b351e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34219 } }
I20260926 03:31:45.570429 5508 raft_consensus.cc:374] T de911a53b444427083ec1a1126da9834 P 0df5da17a0164499b9b20cc1824c6e90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:45.570529 5509 raft_consensus.cc:374] T de911a53b444427083ec1a1126da9834 P e12f413a8222496194fe5ca8b8ddc040 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:45.570580 5508 raft_consensus.cc:729] T de911a53b444427083ec1a1126da9834 P 0df5da17a0164499b9b20cc1824c6e90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0df5da17a0164499b9b20cc1824c6e90, State: Initialized, Role: FOLLOWER
I20260926 03:31:45.570670 5509 raft_consensus.cc:729] T de911a53b444427083ec1a1126da9834 P e12f413a8222496194fe5ca8b8ddc040 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e12f413a8222496194fe5ca8b8ddc040, State: Initialized, Role: FOLLOWER
I20260926 03:31:45.571382 5509 consensus_queue.cc:260] T de911a53b444427083ec1a1126da9834 P e12f413a8222496194fe5ca8b8ddc040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12f413a8222496194fe5ca8b8ddc040" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37719 } } peers { permanent_uuid: "0df5da17a0164499b9b20cc1824c6e90" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32989 } } peers { permanent_uuid: "54aa4c9ccd8c4b309f3d42b5f16b351e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34219 } }
I20260926 03:31:45.572445 5508 consensus_queue.cc:260] T de911a53b444427083ec1a1126da9834 P 0df5da17a0164499b9b20cc1824c6e90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12f413a8222496194fe5ca8b8ddc040" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37719 } } peers { permanent_uuid: "0df5da17a0164499b9b20cc1824c6e90" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32989 } } peers { permanent_uuid: "54aa4c9ccd8c4b309f3d42b5f16b351e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34219 } }
W20260926 03:31:45.575994 5230 tablet.cc:2369] T de911a53b444427083ec1a1126da9834 P 54aa4c9ccd8c4b309f3d42b5f16b351e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:31:45.578729 5514 leader_election.cc:288] T de911a53b444427083ec1a1126da9834 P 54aa4c9ccd8c4b309f3d42b5f16b351e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e12f413a8222496194fe5ca8b8ddc040 (127.0.97.3:37719), 0df5da17a0164499b9b20cc1824c6e90 (127.0.97.2:32989)
I20260926 03:31:45.580804 5509 ts_tablet_manager.cc:1428] T de911a53b444427083ec1a1126da9834 P e12f413a8222496194fe5ca8b8ddc040: Time spent starting tablet: real 0.022s user 0.013s sys 0.010s
I20260926 03:31:45.581749 5490 heartbeater.cc:503] Master 127.0.97.62:45481 was elected leader, sending a full tablet report...
I20260926 03:31:45.588856 5360 heartbeater.cc:503] Master 127.0.97.62:45481 was elected leader, sending a full tablet report...
I20260926 03:31:45.592344 5508 ts_tablet_manager.cc:1428] T de911a53b444427083ec1a1126da9834 P 0df5da17a0164499b9b20cc1824c6e90: Time spent starting tablet: real 0.034s user 0.029s sys 0.000s
I20260926 03:31:45.593535 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:45.589539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40995 (local address 127.0.97.2:32989)
0926 03:31:45.589770 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:45.589777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:45.590567 (+ 790us) server_negotiation.cc:407] Connection header received
0926 03:31:45.590839 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:45.590845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:45.590907 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:45.590992 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:45.592739 (+ 1747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:45.592745 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:45.592750 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:31:45.592770 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:45.592807 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:31:45.592822 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:45.592912 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:45.593189 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 03:31:45.593263 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 03:31:45.593950 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:45.589712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32989 (local address 127.0.97.1:40995)
0926 03:31:45.590465 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:45.590487 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:45.590665 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:45.591225 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:45.591230 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:45.591637 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:45.592460 (+ 823us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:31:45.592507 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:45.592526 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:31:45.592585 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:45.592997 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:45.593002 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:45.593007 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 03:31:45.593162 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 03:31:45.593223 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":318,"threads_started":1}
I20260926 03:31:45.595029 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:45.572142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33445 (local address 127.0.97.3:37719)
0926 03:31:45.572327 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:45.572333 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:45.589220 (+ 16887us) server_negotiation.cc:407] Connection header received
0926 03:31:45.590427 (+ 1207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:45.590433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:45.590499 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:45.590569 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:45.593166 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:45.593173 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:45.593178 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:31:45.593199 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:45.593253 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:31:45.593271 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:45.593374 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:45.594785 (+ 1411us) server_negotiation.cc:299] Negotiation successful
0926 03:31:45.594848 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:31:45.595024 5315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de911a53b444427083ec1a1126da9834" candidate_uuid: "54aa4c9ccd8c4b309f3d42b5f16b351e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0df5da17a0164499b9b20cc1824c6e90" is_pre_election: true
I20260926 03:31:45.595584 5315 raft_consensus.cc:2454] T de911a53b444427083ec1a1126da9834 P 0df5da17a0164499b9b20cc1824c6e90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54aa4c9ccd8c4b309f3d42b5f16b351e in term 0.
I20260926 03:31:45.595923 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:45.572940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37719 (local address 127.0.97.1:33445)
0926 03:31:45.589105 (+ 16165us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:45.589127 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:45.590257 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:45.590775 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:45.590780 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:45.590812 (+ 32us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:45.592367 (+ 1555us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:31:45.592373 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:45.592402 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:31:45.592461 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:45.594636 (+ 2175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:45.594643 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:45.594649 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 03:31:45.595480 (+ 831us) client_negotiation.cc:240] Negotiation successful
0926 03:31:45.595524 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16009,"thread_start_us":15805,"threads_started":1}
I20260926 03:31:45.596309 5117 leader_election.cc:302] T de911a53b444427083ec1a1126da9834 P 54aa4c9ccd8c4b309f3d42b5f16b351e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0df5da17a0164499b9b20cc1824c6e90, 54aa4c9ccd8c4b309f3d42b5f16b351e; no voters:
I20260926 03:31:45.596736 5514 raft_consensus.cc:2781] T de911a53b444427083ec1a1126da9834 P 54aa4c9ccd8c4b309f3d42b5f16b351e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:31:45.596910 5514 raft_consensus.cc:482] T de911a53b444427083ec1a1126da9834 P 54aa4c9ccd8c4b309f3d42b5f16b351e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:31:45.597024 5514 raft_consensus.cc:3037] T de911a53b444427083ec1a1126da9834 P 54aa4c9ccd8c4b309f3d42b5f16b351e [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:45.597370 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de911a53b444427083ec1a1126da9834" candidate_uuid: "54aa4c9ccd8c4b309f3d42b5f16b351e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e12f413a8222496194fe5ca8b8ddc040" is_pre_election: true
I20260926 03:31:45.597800 5445 raft_consensus.cc:2454] T de911a53b444427083ec1a1126da9834 P e12f413a8222496194fe5ca8b8ddc040 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54aa4c9ccd8c4b309f3d42b5f16b351e in term 0.
I20260926 03:31:45.599774 5514 raft_consensus.cc:504] T de911a53b444427083ec1a1126da9834 P 54aa4c9ccd8c4b309f3d42b5f16b351e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12f413a8222496194fe5ca8b8ddc040" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37719 } } peers { permanent_uuid: "0df5da17a0164499b9b20cc1824c6e90" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32989 } } peers { permanent_uuid: "54aa4c9ccd8c4b309f3d42b5f16b351e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34219 } }
I20260926 03:31:45.600673 5514 leader_election.cc:288] T de911a53b444427083ec1a1126da9834 P 54aa4c9ccd8c4b309f3d42b5f16b351e [CANDIDATE]: Term 1 election: Requested vote from peers e12f413a8222496194fe5ca8b8ddc040 (127.0.97.3:37719), 0df5da17a0164499b9b20cc1824c6e90 (127.0.97.2:32989)
I20260926 03:31:45.601143 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de911a53b444427083ec1a1126da9834" candidate_uuid: "54aa4c9ccd8c4b309f3d42b5f16b351e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e12f413a8222496194fe5ca8b8ddc040"
I20260926 03:31:45.601182 5315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de911a53b444427083ec1a1126da9834" candidate_uuid: "54aa4c9ccd8c4b309f3d42b5f16b351e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0df5da17a0164499b9b20cc1824c6e90"
I20260926 03:31:45.601400 5315 raft_consensus.cc:3037] T de911a53b444427083ec1a1126da9834 P 0df5da17a0164499b9b20cc1824c6e90 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:45.601401 5445 raft_consensus.cc:3037] T de911a53b444427083ec1a1126da9834 P e12f413a8222496194fe5ca8b8ddc040 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:45.604969 5315 raft_consensus.cc:2454] T de911a53b444427083ec1a1126da9834 P 0df5da17a0164499b9b20cc1824c6e90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54aa4c9ccd8c4b309f3d42b5f16b351e in term 1.
I20260926 03:31:45.604969 5445 raft_consensus.cc:2454] T de911a53b444427083ec1a1126da9834 P e12f413a8222496194fe5ca8b8ddc040 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54aa4c9ccd8c4b309f3d42b5f16b351e in term 1.
I20260926 03:31:45.605406 5117 leader_election.cc:302] T de911a53b444427083ec1a1126da9834 P 54aa4c9ccd8c4b309f3d42b5f16b351e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54aa4c9ccd8c4b309f3d42b5f16b351e, e12f413a8222496194fe5ca8b8ddc040; no voters:
I20260926 03:31:45.605686 5514 raft_consensus.cc:2781] T de911a53b444427083ec1a1126da9834 P 54aa4c9ccd8c4b309f3d42b5f16b351e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:31:45.606204 5514 raft_consensus.cc:686] T de911a53b444427083ec1a1126da9834 P 54aa4c9ccd8c4b309f3d42b5f16b351e [term 1 LEADER]: Becoming Leader. State: Replica: 54aa4c9ccd8c4b309f3d42b5f16b351e, State: Running, Role: LEADER
I20260926 03:31:45.606734 5514 consensus_queue.cc:237] T de911a53b444427083ec1a1126da9834 P 54aa4c9ccd8c4b309f3d42b5f16b351e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12f413a8222496194fe5ca8b8ddc040" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37719 } } peers { permanent_uuid: "0df5da17a0164499b9b20cc1824c6e90" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32989 } } peers { permanent_uuid: "54aa4c9ccd8c4b309f3d42b5f16b351e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34219 } }
I20260926 03:31:45.611961 5043 catalog_manager.cc:5495] T de911a53b444427083ec1a1126da9834 P 54aa4c9ccd8c4b309f3d42b5f16b351e reported cstate change: term changed from 0 to 1, leader changed from <none> to 54aa4c9ccd8c4b309f3d42b5f16b351e (127.0.97.1). New cstate: current_term: 1 leader_uuid: "54aa4c9ccd8c4b309f3d42b5f16b351e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e12f413a8222496194fe5ca8b8ddc040" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0df5da17a0164499b9b20cc1824c6e90" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54aa4c9ccd8c4b309f3d42b5f16b351e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34219 } health_report { overall_health: HEALTHY } } }
I20260926 03:31:45.629514 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:45.627296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53026 (local address 127.0.97.1:34219)
0926 03:31:45.627474 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:45.627479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:45.627498 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:31:45.627704 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:45.627708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:45.627779 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:45.627868 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:45.628896 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:45.628900 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:45.628903 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:31:45.628916 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:45.628944 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:31:45.628955 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:45.629055 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:45.629159 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 03:31:45.629206 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:31:45.639076 5445 raft_consensus.cc:1261] T de911a53b444427083ec1a1126da9834 P e12f413a8222496194fe5ca8b8ddc040 [term 1 FOLLOWER]: Refusing update from remote peer 54aa4c9ccd8c4b309f3d42b5f16b351e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:31:45.639078 5315 raft_consensus.cc:1261] T de911a53b444427083ec1a1126da9834 P 0df5da17a0164499b9b20cc1824c6e90 [term 1 FOLLOWER]: Refusing update from remote peer 54aa4c9ccd8c4b309f3d42b5f16b351e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:31:45.640170 5519 consensus_queue.cc:1035] T de911a53b444427083ec1a1126da9834 P 54aa4c9ccd8c4b309f3d42b5f16b351e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e12f413a8222496194fe5ca8b8ddc040" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:31:45.640585 5514 consensus_queue.cc:1035] T de911a53b444427083ec1a1126da9834 P 54aa4c9ccd8c4b309f3d42b5f16b351e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0df5da17a0164499b9b20cc1824c6e90" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:31:45.655890 5522 mvcc.cc:205] Tried to move back new op lower bound from 7333451799085776896 to 7333451798969544704. Current Snapshot: MvccSnapshot[applied={T|T < 7333451799085776896}]
I20260926 03:31:45.673663 5042 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:31:45.673935 5042 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:31:45.677742 5042 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d32560a13e09469ea9a03ad892696ae2: Sending DeleteTablet for 3 replicas of tablet de911a53b444427083ec1a1126da9834
I20260926 03:31:45.679090 5295 tablet_service.cc:1514] Processing DeleteTablet for tablet de911a53b444427083ec1a1126da9834 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:31:45 UTC) from {username='slave'} at 127.0.0.1:46920
I20260926 03:31:45.679319 5425 tablet_service.cc:1514] Processing DeleteTablet for tablet de911a53b444427083ec1a1126da9834 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:31:45 UTC) from {username='slave'} at 127.0.0.1:50802
I20260926 03:31:45.679278 5164 tablet_service.cc:1514] Processing DeleteTablet for tablet de911a53b444427083ec1a1126da9834 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:31:45 UTC) from {username='slave'} at 127.0.0.1:53010
I20260926 03:31:45.683744 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 5102
I20260926 03:31:45.698282 5531 tablet_replica.cc:330] T de911a53b444427083ec1a1126da9834 P 0df5da17a0164499b9b20cc1824c6e90: stopping tablet replica
I20260926 03:31:45.698457 5533 tablet_replica.cc:330] T de911a53b444427083ec1a1126da9834 P 54aa4c9ccd8c4b309f3d42b5f16b351e: stopping tablet replica
I20260926 03:31:45.698880 5533 raft_consensus.cc:2229] T de911a53b444427083ec1a1126da9834 P 54aa4c9ccd8c4b309f3d42b5f16b351e [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:31:45.698880 5531 raft_consensus.cc:2229] T de911a53b444427083ec1a1126da9834 P 0df5da17a0164499b9b20cc1824c6e90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:31:45.699505 5531 raft_consensus.cc:2258] T de911a53b444427083ec1a1126da9834 P 0df5da17a0164499b9b20cc1824c6e90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:31:45.699635 5533 raft_consensus.cc:2258] T de911a53b444427083ec1a1126da9834 P 54aa4c9ccd8c4b309f3d42b5f16b351e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:31:45.701351 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:45.686210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53028 (local address 127.0.97.1:34219)
0926 03:31:45.686419 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:45.686426 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:45.696048 (+ 9622us) server_negotiation.cc:407] Connection header received
0926 03:31:45.696637 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:45.696648 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:45.696765 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:45.696878 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:45.700347 (+ 3469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:45.700354 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:45.700360 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:31:45.700381 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:45.700426 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:31:45.700444 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:45.700551 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:45.700866 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 03:31:45.700960 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:31:45.702559 5224 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:31:45.703444 5532 tablet_replica.cc:330] T de911a53b444427083ec1a1126da9834 P e12f413a8222496194fe5ca8b8ddc040: stopping tablet replica
I20260926 03:31:45.704109 5532 raft_consensus.cc:2229] T de911a53b444427083ec1a1126da9834 P e12f413a8222496194fe5ca8b8ddc040 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:31:45.704741 5531 ts_tablet_manager.cc:1905] T de911a53b444427083ec1a1126da9834 P 0df5da17a0164499b9b20cc1824c6e90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:31:45.704854 5532 raft_consensus.cc:2258] T de911a53b444427083ec1a1126da9834 P e12f413a8222496194fe5ca8b8ddc040 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:31:45.707870 5532 ts_tablet_manager.cc:1905] T de911a53b444427083ec1a1126da9834 P e12f413a8222496194fe5ca8b8ddc040: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:31:45.710616 5531 ts_tablet_manager.cc:1918] T de911a53b444427083ec1a1126da9834 P 0df5da17a0164499b9b20cc1824c6e90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:31:45.710824 5531 log.cc:1196] T de911a53b444427083ec1a1126da9834 P 0df5da17a0164499b9b20cc1824c6e90: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-1/wal/wals/de911a53b444427083ec1a1126da9834
I20260926 03:31:45.711359 5531 ts_tablet_manager.cc:1939] T de911a53b444427083ec1a1126da9834 P 0df5da17a0164499b9b20cc1824c6e90: Deleting consensus metadata
I20260926 03:31:45.713349 5027 catalog_manager.cc:4841] TS 0df5da17a0164499b9b20cc1824c6e90 (127.0.97.2:32989): tablet de911a53b444427083ec1a1126da9834 (table test-table [id=3599f079c9c840bca81cbac2ca2ff0ab]) successfully deleted
I20260926 03:31:45.717864 5532 ts_tablet_manager.cc:1918] T de911a53b444427083ec1a1126da9834 P e12f413a8222496194fe5ca8b8ddc040: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:31:45.718036 5532 log.cc:1196] T de911a53b444427083ec1a1126da9834 P e12f413a8222496194fe5ca8b8ddc040: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790393463219689-388-0/minicluster-data/ts-2/wal/wals/de911a53b444427083ec1a1126da9834
I20260926 03:31:45.718542 5532 ts_tablet_manager.cc:1939] T de911a53b444427083ec1a1126da9834 P e12f413a8222496194fe5ca8b8ddc040: Deleting consensus metadata
I20260926 03:31:45.720237 5027 catalog_manager.cc:4841] TS e12f413a8222496194fe5ca8b8ddc040 (127.0.97.3:37719): tablet de911a53b444427083ec1a1126da9834 (table test-table [id=3599f079c9c840bca81cbac2ca2ff0ab]) successfully deleted
I20260926 03:31:45.703414 5533 ts_tablet_manager.cc:1905] T de911a53b444427083ec1a1126da9834 P 54aa4c9ccd8c4b309f3d42b5f16b351e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:31:45.805971 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 5102
W20260926 03:31:45.826956 5030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:34219 (error 108)
I20260926 03:31:45.827050 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 5233
I20260926 03:31:45.828975 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:45.828523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34219 (local address 127.0.0.1:53044)
0926 03:31:45.828726 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:45.828833 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260926 03:31:45.829303 5030 catalog_manager.cc:4584] TS 54aa4c9ccd8c4b309f3d42b5f16b351e (127.0.97.1:34219): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet de911a53b444427083ec1a1126da9834: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34219: connect: Connection refused (error 111)
I20260926 03:31:45.830654 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:45.827976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46932 (local address 127.0.97.2:32989)
0926 03:31:45.828180 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:45.828186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:45.828271 (+ 85us) server_negotiation.cc:407] Connection header received
0926 03:31:45.828523 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:45.828530 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:45.828699 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:45.828857 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:45.829976 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:45.829982 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:45.829993 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:31:45.830010 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:45.830043 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:31:45.830055 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:45.830189 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:45.830277 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 03:31:45.830347 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:31:45.831383 5355 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:31:45.883457 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:45.882870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34219 (local address 127.0.0.1:53060)
0926 03:31:45.883156 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:45.883277 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 03:31:45.884502 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:45.884199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34219 (local address 127.0.0.1:53076)
0926 03:31:45.884334 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:45.884408 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 03:31:45.933097 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 5233
I20260926 03:31:45.945210 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:45.944725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34219 (local address 127.0.0.1:53090)
0926 03:31:45.944937 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:45.945041 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 03:31:45.946370 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:45.946027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34219 (local address 127.0.0.1:53096)
0926 03:31:45.946181 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:45.946251 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 03:31:45.954339 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 5363
I20260926 03:31:45.957647 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:45.955304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50818 (local address 127.0.97.3:37719)
0926 03:31:45.955481 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:45.955486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:45.955590 (+ 104us) server_negotiation.cc:407] Connection header received
0926 03:31:45.955772 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:45.955776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:45.955860 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:45.955977 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:45.957027 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:45.957032 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:45.957035 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:31:45.957050 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:45.957082 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:31:45.957093 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:45.957203 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:45.957371 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 03:31:45.957420 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 03:31:45.958344 5485 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:31:46.027151 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:46.026536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34219 (local address 127.0.0.1:53112)
0926 03:31:46.026799 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:46.026924 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260926 03:31:46.028533 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:46.028116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34219 (local address 127.0.0.1:53118)
0926 03:31:46.028349 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:46.028427 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260926 03:31:46.052421 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 5363
I20260926 03:31:46.073297 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 5012
I20260926 03:31:46.074558 5073 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:31:46.188994 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 5012
2026-09-26T03:31:46Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2610 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 03:31:46.232374 388 mini_oidc.cc:152] Starting JWKS server
I20260926 03:31:46.234980 388 webserver.cc:460] Webserver started at https://127.0.0.1:34585/ using document root <none> and password file <none>
I20260926 03:31:46.235231 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 03:31:46.235838 388 webserver.cc:460] Webserver started at http://127.0.0.1:46531/ using document root <none> and password file <none>
2026-09-26T03:31:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:31:46Z Disabled control of system clock
I20260926 03:31:46.262850 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45583
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:43353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45583
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34585/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393463219689-388-0/testchainca.cert with env {}
W20260926 03:31:46.585630 5545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:46.585901 5545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:46.586066 5545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:46.593165 5545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:31:46.593279 5545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:46.593344 5545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:46.593446 5545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:31:46.593537 5545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 03:31:46.593647 5545 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 03:31:46.593722 5545 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34585/jwks.json
I20260926 03:31:46.603253 5545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45583
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34585/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393463219689-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:46.604540 5545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:46.605847 5545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:46.617331 5550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:46.617290 5551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:46.617264 5553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:46.618005 5545 server_base.cc:1034] running on GCE node
I20260926 03:31:46.618717 5545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:46.619939 5545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:46.621155 5545 hybrid_clock.cc:648] HybridClock initialized: now 1790393506621108 us; error 37 us; skew 500 ppm
I20260926 03:31:46.621485 5545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:31:46.635383 5545 webserver.cc:460] Webserver started at http://127.0.97.62:36769/ using document root <none> and password file <none>
I20260926 03:31:46.636200 5545 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:46.636340 5545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:46.636754 5545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:46.639793 5545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "4595bf1851a745c18528ff1b4371aece"
format_stamp: "Formatted at 2026-09-26 03:31:46 on dist-test-slave-dt7p"
server_key: "d6d47e5d4c2766c606e9e98f816f73f4"
server_key_iv: "d3fa45fc7ccf0f7f89731c2c2fe9a644"
server_key_version: "encryptionkey@0"
I20260926 03:31:46.640671 5545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "4595bf1851a745c18528ff1b4371aece"
format_stamp: "Formatted at 2026-09-26 03:31:46 on dist-test-slave-dt7p"
server_key: "d6d47e5d4c2766c606e9e98f816f73f4"
server_key_iv: "d3fa45fc7ccf0f7f89731c2c2fe9a644"
server_key_version: "encryptionkey@0"
I20260926 03:31:46.645603 5545 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 03:31:46.648929 5559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:46.650193 5545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:31:46.650370 5545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "4595bf1851a745c18528ff1b4371aece"
format_stamp: "Formatted at 2026-09-26 03:31:46 on dist-test-slave-dt7p"
server_key: "d6d47e5d4c2766c606e9e98f816f73f4"
server_key_iv: "d3fa45fc7ccf0f7f89731c2c2fe9a644"
server_key_version: "encryptionkey@0"
I20260926 03:31:46.650588 5545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:46.675051 5545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:46.675902 5545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:46.676193 5545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:46.694471 5545 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45583
I20260926 03:31:46.694490 5610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45583 every 8 connection(s)
I20260926 03:31:46.695772 5545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:31:46.699469 5611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:46.701241 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 5545
I20260926 03:31:46.701436 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:31:46.702060 388 external_mini_cluster.cc:1428] Setting key fcfe5477660d4cec2cc3c3a5ab4559de
I20260926 03:31:46.710866 5611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4595bf1851a745c18528ff1b4371aece: Bootstrap starting.
I20260926 03:31:46.714784 5611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4595bf1851a745c18528ff1b4371aece: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:46.716150 5614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:46.703919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54364 (local address 127.0.97.62:45583)
0926 03:31:46.704933 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:46.704943 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:46.704986 (+ 43us) server_negotiation.cc:407] Connection header received
0926 03:31:46.706563 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:46.706583 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:46.706985 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:46.707297 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:46.707906 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:46.709268 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:46.711123 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:46.711733 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:46.711854 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:46.713287 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:46.713297 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:46.713314 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:31:46.713982 (+ 668us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:46.714103 (+ 121us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:31:46.714131 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:46.714247 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:46.714485 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 03:31:46.714858 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":333,"threads_started":1}
I20260926 03:31:46.716187 5611 log.cc:824] T 00000000000000000000000000000000 P 4595bf1851a745c18528ff1b4371aece: Log is configured to *not* fsync() on all Append() calls
I20260926 03:31:46.719412 5611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4595bf1851a745c18528ff1b4371aece: No bootstrap required, opened a new log
I20260926 03:31:46.720335 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:31:46.727814 5611 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4595bf1851a745c18528ff1b4371aece [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4595bf1851a745c18528ff1b4371aece" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45583 } }
I20260926 03:31:46.728123 5611 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4595bf1851a745c18528ff1b4371aece [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:46.728193 5611 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4595bf1851a745c18528ff1b4371aece [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4595bf1851a745c18528ff1b4371aece, State: Initialized, Role: FOLLOWER
I20260926 03:31:46.728775 5611 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4595bf1851a745c18528ff1b4371aece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4595bf1851a745c18528ff1b4371aece" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45583 } }
I20260926 03:31:46.728964 5611 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4595bf1851a745c18528ff1b4371aece [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:31:46.729059 5611 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4595bf1851a745c18528ff1b4371aece [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:31:46.729173 5611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4595bf1851a745c18528ff1b4371aece [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:46.731328 5611 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4595bf1851a745c18528ff1b4371aece [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4595bf1851a745c18528ff1b4371aece" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45583 } }
I20260926 03:31:46.731765 5611 leader_election.cc:302] T 00000000000000000000000000000000 P 4595bf1851a745c18528ff1b4371aece [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4595bf1851a745c18528ff1b4371aece; no voters:
I20260926 03:31:46.732280 5611 leader_election.cc:288] T 00000000000000000000000000000000 P 4595bf1851a745c18528ff1b4371aece [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:31:46.732439 5616 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4595bf1851a745c18528ff1b4371aece [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:31:46.733144 5616 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4595bf1851a745c18528ff1b4371aece [term 1 LEADER]: Becoming Leader. State: Replica: 4595bf1851a745c18528ff1b4371aece, State: Running, Role: LEADER
I20260926 03:31:46.733690 5616 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4595bf1851a745c18528ff1b4371aece [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4595bf1851a745c18528ff1b4371aece" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45583 } }
I20260926 03:31:46.734261 5611 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4595bf1851a745c18528ff1b4371aece [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:31:46.736642 5618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4595bf1851a745c18528ff1b4371aece [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4595bf1851a745c18528ff1b4371aece. Latest consensus state: current_term: 1 leader_uuid: "4595bf1851a745c18528ff1b4371aece" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4595bf1851a745c18528ff1b4371aece" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45583 } } }
I20260926 03:31:46.737079 5618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4595bf1851a745c18528ff1b4371aece [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:46.736630 5617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4595bf1851a745c18528ff1b4371aece [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4595bf1851a745c18528ff1b4371aece" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4595bf1851a745c18528ff1b4371aece" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45583 } } }
I20260926 03:31:46.737267 5617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4595bf1851a745c18528ff1b4371aece [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:46.737988 5625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:31:46.743755 5625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:31:46.753391 5625 catalog_manager.cc:1347] Generated new cluster ID: 52b247f7c66b47fc87cd5e7e218977e5
I20260926 03:31:46.753523 5625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:31:46.767915 5625 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:31:46.769603 5625 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:31:46.781456 5625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4595bf1851a745c18528ff1b4371aece: Generated new TSK 0
I20260926 03:31:46.782509 5625 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 03:31:46.820747 5641 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 03:31:46.820895 5641 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 03:31:46.822033 5614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:46.800727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54376 (local address 127.0.97.62:45583)
0926 03:31:46.800925 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:46.800932 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:46.801514 (+ 582us) server_negotiation.cc:407] Connection header received
0926 03:31:46.801763 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:46.801768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:46.801852 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:46.802007 (+ 155us) server_negotiation.cc:226] Negotiated authn=JWT
0926 03:31:46.802488 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:46.803621 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:46.805040 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:46.805707 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:46.805847 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:46.806288 (+ 441us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 03:31:46.821413 (+ 15125us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34585/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 03:31:46.821735 (+ 322us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54376: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34585/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 03:31:46.822155 5614 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54376: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34585/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 03:31:46.823280 5640 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:31:46.800781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45583 (local address 127.0.0.1:54376)
0926 03:31:46.801409 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:46.801424 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:46.801560 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:46.802023 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:46.802026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:46.802056 (+ 30us) client_negotiation.cc:189] Negotiated authn=JWT
0926 03:31:46.802354 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:46.802362 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:46.803747 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:46.803751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:46.804907 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:46.804918 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:46.805820 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:46.805867 (+ 47us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:46.805979 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:46.806041 (+ 62us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 03:31:46.821665 (+ 15624us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34585/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 03:31:46.821894 (+ 229us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:45583: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34585/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":341,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:45583: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34585/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 03:31:46.824717 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 5545
I20260926 03:31:46.825780 5606 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:31:46.916824 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 5545
2026-09-26T03:31:46Z chronyd exiting
I20260926 03:31:46.966831 388 test_util.cc:175] -----------------------------------------------
I20260926 03:31:46.966945 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790393463219689-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (738 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 03:31:46.970497 388 mini_oidc.cc:152] Starting JWKS server
I20260926 03:31:46.972180 388 webserver.cc:460] Webserver started at https://127.0.0.1:46213/ using document root <none> and password file <none>
I20260926 03:31:46.972407 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 03:31:46.973034 388 webserver.cc:460] Webserver started at http://127.0.0.1:39657/ using document root <none> and password file <none>
2026-09-26T03:31:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:31:46Z Disabled control of system clock
I20260926 03:31:46.997850 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41393
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:44859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41393
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46213/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790393463219689-388-0/testchainca.cert with env {}
W20260926 03:31:47.309779 5648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:47.310051 5648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:47.310137 5648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:47.317283 5648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:31:47.317382 5648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:47.317420 5648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:47.317467 5648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:31:47.317509 5648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 03:31:47.317579 5648 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 03:31:47.317611 5648 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46213/jwks.json
I20260926 03:31:47.327176 5648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41393
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46213/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790393463219689-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:47.328308 5648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:47.329488 5648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:47.340951 5656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:47.340997 5654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:47.341001 5653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:47.341652 5648 server_base.cc:1034] running on GCE node
I20260926 03:31:47.342413 5648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:47.343706 5648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:47.344934 5648 hybrid_clock.cc:648] HybridClock initialized: now 1790393507344886 us; error 61 us; skew 500 ppm
I20260926 03:31:47.345283 5648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:31:47.347467 5648 webserver.cc:460] Webserver started at http://127.0.97.62:38105/ using document root <none> and password file <none>
I20260926 03:31:47.348132 5648 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:47.348260 5648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:47.348610 5648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:47.351354 5648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "6a477a6af6f645cf91ee59fdca705a88"
format_stamp: "Formatted at 2026-09-26 03:31:47 on dist-test-slave-dt7p"
server_key: "0224bcb2ee447d73abc849d518a0a5cf"
server_key_iv: "a426014629bf11c186fbd89cc75de125"
server_key_version: "encryptionkey@0"
I20260926 03:31:47.352130 5648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "6a477a6af6f645cf91ee59fdca705a88"
format_stamp: "Formatted at 2026-09-26 03:31:47 on dist-test-slave-dt7p"
server_key: "0224bcb2ee447d73abc849d518a0a5cf"
server_key_iv: "a426014629bf11c186fbd89cc75de125"
server_key_version: "encryptionkey@0"
I20260926 03:31:47.356923 5648 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 03:31:47.360162 5662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:47.361550 5648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:31:47.361745 5648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "6a477a6af6f645cf91ee59fdca705a88"
format_stamp: "Formatted at 2026-09-26 03:31:47 on dist-test-slave-dt7p"
server_key: "0224bcb2ee447d73abc849d518a0a5cf"
server_key_iv: "a426014629bf11c186fbd89cc75de125"
server_key_version: "encryptionkey@0"
I20260926 03:31:47.361974 5648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:47.400370 5648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:47.401273 5648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:47.401566 5648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:47.419567 5713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41393 every 8 connection(s)
I20260926 03:31:47.419543 5648 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41393
I20260926 03:31:47.421069 5648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:31:47.424870 5714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:47.425037 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 5648
I20260926 03:31:47.425247 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:31:47.425859 388 external_mini_cluster.cc:1428] Setting key 280e9698c46e575981e263ff328a8fe5
I20260926 03:31:47.439438 5714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a477a6af6f645cf91ee59fdca705a88: Bootstrap starting.
I20260926 03:31:47.441851 5717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:47.427820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44244 (local address 127.0.97.62:41393)
0926 03:31:47.428922 (+ 1102us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:47.428935 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:47.428985 (+ 50us) server_negotiation.cc:407] Connection header received
0926 03:31:47.431418 (+ 2433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:47.431446 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:47.432039 (+ 593us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:47.432472 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:47.433016 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:47.434884 (+ 1868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:47.436128 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:47.437393 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:47.437589 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:47.438784 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:47.438795 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:47.438815 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:31:47.439555 (+ 740us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:47.439688 (+ 133us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:31:47.439731 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:47.439896 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:47.440168 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 03:31:47.440504 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":337,"threads_started":1}
I20260926 03:31:47.443554 5714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a477a6af6f645cf91ee59fdca705a88: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:47.444774 5714 log.cc:824] T 00000000000000000000000000000000 P 6a477a6af6f645cf91ee59fdca705a88: Log is configured to *not* fsync() on all Append() calls
I20260926 03:31:47.445859 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:31:47.448012 5714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a477a6af6f645cf91ee59fdca705a88: No bootstrap required, opened a new log
I20260926 03:31:47.455435 5714 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a477a6af6f645cf91ee59fdca705a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a477a6af6f645cf91ee59fdca705a88" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41393 } }
I20260926 03:31:47.455766 5714 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a477a6af6f645cf91ee59fdca705a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:47.455840 5714 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a477a6af6f645cf91ee59fdca705a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a477a6af6f645cf91ee59fdca705a88, State: Initialized, Role: FOLLOWER
I20260926 03:31:47.456348 5714 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a477a6af6f645cf91ee59fdca705a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a477a6af6f645cf91ee59fdca705a88" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41393 } }
I20260926 03:31:47.456697 5714 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a477a6af6f645cf91ee59fdca705a88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:31:47.456858 5714 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a477a6af6f645cf91ee59fdca705a88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:31:47.457047 5714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a477a6af6f645cf91ee59fdca705a88 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:47.459123 5714 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a477a6af6f645cf91ee59fdca705a88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a477a6af6f645cf91ee59fdca705a88" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41393 } }
I20260926 03:31:47.459614 5714 leader_election.cc:302] T 00000000000000000000000000000000 P 6a477a6af6f645cf91ee59fdca705a88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a477a6af6f645cf91ee59fdca705a88; no voters:
I20260926 03:31:47.460143 5714 leader_election.cc:288] T 00000000000000000000000000000000 P 6a477a6af6f645cf91ee59fdca705a88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:31:47.460286 5719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a477a6af6f645cf91ee59fdca705a88 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:31:47.461052 5719 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a477a6af6f645cf91ee59fdca705a88 [term 1 LEADER]: Becoming Leader. State: Replica: 6a477a6af6f645cf91ee59fdca705a88, State: Running, Role: LEADER
I20260926 03:31:47.461592 5719 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a477a6af6f645cf91ee59fdca705a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a477a6af6f645cf91ee59fdca705a88" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41393 } }
I20260926 03:31:47.462188 5714 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a477a6af6f645cf91ee59fdca705a88 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:31:47.464555 5721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a477a6af6f645cf91ee59fdca705a88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a477a6af6f645cf91ee59fdca705a88. Latest consensus state: current_term: 1 leader_uuid: "6a477a6af6f645cf91ee59fdca705a88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a477a6af6f645cf91ee59fdca705a88" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41393 } } }
I20260926 03:31:47.464995 5721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a477a6af6f645cf91ee59fdca705a88 [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:47.466008 5720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a477a6af6f645cf91ee59fdca705a88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a477a6af6f645cf91ee59fdca705a88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a477a6af6f645cf91ee59fdca705a88" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41393 } } }
I20260926 03:31:47.466349 5720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a477a6af6f645cf91ee59fdca705a88 [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:47.470268 5725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:31:47.475488 5725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:31:47.483405 5725 catalog_manager.cc:1347] Generated new cluster ID: 2a5b016d04124956abdb15277dffe618
I20260926 03:31:47.483522 5725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:31:47.494771 5725 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:31:47.496479 5725 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:31:47.513206 5725 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a477a6af6f645cf91ee59fdca705a88: Generated new TSK 0
I20260926 03:31:47.514209 5725 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 03:31:47.556102 5744 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 03:31:47.556246 5744 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260926 03:31:47.557513 5743 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:31:47.537317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41393 (local address 127.0.0.1:44258)
0926 03:31:47.538025 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:47.538044 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:47.538159 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:47.538566 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:47.538569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:47.538578 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0926 03:31:47.538799 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:47.538807 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:47.540217 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:47.540220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:47.541363 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:47.541374 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:47.542311 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:47.542350 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:47.542530 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:47.542563 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 03:31:47.557034 (+ 14471us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46213/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 03:31:47.557221 (+ 187us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:41393: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46213/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":427,"threads_started":1}
I20260926 03:31:47.557518 5717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:47.537459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44258 (local address 127.0.97.62:41393)
0926 03:31:47.537646 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:47.537653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:47.538120 (+ 467us) server_negotiation.cc:407] Connection header received
0926 03:31:47.538309 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:47.538314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:47.538397 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:47.538554 (+ 157us) server_negotiation.cc:226] Negotiated authn=JWT
0926 03:31:47.538934 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:47.540086 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:47.541491 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:47.542172 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:47.542295 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:47.542828 (+ 533us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 03:31:47.556806 (+ 13978us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46213/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 03:31:47.557198 (+ 392us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44258: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46213/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 03:31:47.557644 5717 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44258: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46213/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260926 03:31:47.559043 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 5648
I20260926 03:31:47.560137 5709 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:31:47.653690 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 5648
2026-09-26T03:31:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (740 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 03:31:47.710855 388 mini_oidc.cc:152] Starting JWKS server
I20260926 03:31:47.712690 388 webserver.cc:460] Webserver started at https://127.0.0.1:34627/ using document root <none> and password file <none>
I20260926 03:31:47.712891 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 03:31:47.713464 388 webserver.cc:460] Webserver started at http://127.0.0.1:37091/ using document root <none> and password file <none>
2026-09-26T03:31:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:31:47Z Disabled control of system clock
I20260926 03:31:47.741081 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42045
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:40377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42045
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34627/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 03:31:48.057729 5751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:48.058089 5751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:48.058259 5751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:48.065582 5751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:31:48.065742 5751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:48.065834 5751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:48.065945 5751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:31:48.066038 5751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 03:31:48.066129 5751 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 03:31:48.066208 5751 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34627/jwks.json
I20260926 03:31:48.075397 5751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42045
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34627/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:48.076750 5751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:48.078501 5751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:48.090005 5759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:48.090082 5756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:48.090104 5757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:48.090946 5751 server_base.cc:1034] running on GCE node
I20260926 03:31:48.091641 5751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:48.092991 5751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:48.094198 5751 hybrid_clock.cc:648] HybridClock initialized: now 1790393508094150 us; error 38 us; skew 500 ppm
I20260926 03:31:48.094523 5751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:31:48.096969 5751 webserver.cc:460] Webserver started at http://127.0.97.62:34851/ using document root <none> and password file <none>
I20260926 03:31:48.097606 5751 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:48.097687 5751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:48.097970 5751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:48.100764 5751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "83a4c45092c24c16a723f2ad0407b555"
format_stamp: "Formatted at 2026-09-26 03:31:48 on dist-test-slave-dt7p"
server_key: "5c505ae456985a21c2ba4c78c90c180d"
server_key_iv: "94d9e49a5057ac5bd3dee1898ac8f90d"
server_key_version: "encryptionkey@0"
I20260926 03:31:48.101498 5751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "83a4c45092c24c16a723f2ad0407b555"
format_stamp: "Formatted at 2026-09-26 03:31:48 on dist-test-slave-dt7p"
server_key: "5c505ae456985a21c2ba4c78c90c180d"
server_key_iv: "94d9e49a5057ac5bd3dee1898ac8f90d"
server_key_version: "encryptionkey@0"
I20260926 03:31:48.106424 5751 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260926 03:31:48.109735 5765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:48.111042 5751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:31:48.111217 5751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "83a4c45092c24c16a723f2ad0407b555"
format_stamp: "Formatted at 2026-09-26 03:31:48 on dist-test-slave-dt7p"
server_key: "5c505ae456985a21c2ba4c78c90c180d"
server_key_iv: "94d9e49a5057ac5bd3dee1898ac8f90d"
server_key_version: "encryptionkey@0"
I20260926 03:31:48.111435 5751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:48.124266 5751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:48.125129 5751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:48.125403 5751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:48.142848 5751 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42045
I20260926 03:31:48.142874 5816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42045 every 8 connection(s)
I20260926 03:31:48.144218 5751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:31:48.148175 5817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:48.148697 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 5751
I20260926 03:31:48.148897 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:31:48.149565 388 external_mini_cluster.cc:1428] Setting key 767a70ce7cb2700be8906652e3263227
I20260926 03:31:48.161845 5817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83a4c45092c24c16a723f2ad0407b555: Bootstrap starting.
I20260926 03:31:48.163184 5820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:48.151551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48298 (local address 127.0.97.62:42045)
0926 03:31:48.152610 (+ 1059us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:48.152627 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:48.152678 (+ 51us) server_negotiation.cc:407] Connection header received
0926 03:31:48.154208 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:48.154229 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:48.154671 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:48.154990 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:48.155582 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:48.156994 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:48.158897 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:48.159564 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:48.159693 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:48.160925 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:48.160933 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:48.160947 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:31:48.161470 (+ 523us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:48.161547 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:31:48.161567 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:48.161677 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:48.161898 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 03:31:48.162242 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":371,"threads_started":1}
I20260926 03:31:48.165313 5817 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83a4c45092c24c16a723f2ad0407b555: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:48.166540 5817 log.cc:824] T 00000000000000000000000000000000 P 83a4c45092c24c16a723f2ad0407b555: Log is configured to *not* fsync() on all Append() calls
I20260926 03:31:48.167089 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:31:48.169974 5817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83a4c45092c24c16a723f2ad0407b555: No bootstrap required, opened a new log
I20260926 03:31:48.177287 5817 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83a4c45092c24c16a723f2ad0407b555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a4c45092c24c16a723f2ad0407b555" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42045 } }
I20260926 03:31:48.177680 5817 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83a4c45092c24c16a723f2ad0407b555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:48.177785 5817 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83a4c45092c24c16a723f2ad0407b555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83a4c45092c24c16a723f2ad0407b555, State: Initialized, Role: FOLLOWER
I20260926 03:31:48.178403 5817 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83a4c45092c24c16a723f2ad0407b555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a4c45092c24c16a723f2ad0407b555" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42045 } }
I20260926 03:31:48.178669 5817 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83a4c45092c24c16a723f2ad0407b555 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:31:48.178835 5817 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83a4c45092c24c16a723f2ad0407b555 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:31:48.179003 5817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83a4c45092c24c16a723f2ad0407b555 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:48.181311 5817 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83a4c45092c24c16a723f2ad0407b555 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a4c45092c24c16a723f2ad0407b555" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42045 } }
I20260926 03:31:48.181770 5817 leader_election.cc:302] T 00000000000000000000000000000000 P 83a4c45092c24c16a723f2ad0407b555 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83a4c45092c24c16a723f2ad0407b555; no voters:
I20260926 03:31:48.182353 5817 leader_election.cc:288] T 00000000000000000000000000000000 P 83a4c45092c24c16a723f2ad0407b555 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:31:48.182535 5822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83a4c45092c24c16a723f2ad0407b555 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:31:48.183276 5822 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83a4c45092c24c16a723f2ad0407b555 [term 1 LEADER]: Becoming Leader. State: Replica: 83a4c45092c24c16a723f2ad0407b555, State: Running, Role: LEADER
I20260926 03:31:48.183831 5822 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83a4c45092c24c16a723f2ad0407b555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a4c45092c24c16a723f2ad0407b555" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42045 } }
I20260926 03:31:48.184453 5817 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83a4c45092c24c16a723f2ad0407b555 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:31:48.187361 5824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83a4c45092c24c16a723f2ad0407b555 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83a4c45092c24c16a723f2ad0407b555. Latest consensus state: current_term: 1 leader_uuid: "83a4c45092c24c16a723f2ad0407b555" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a4c45092c24c16a723f2ad0407b555" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42045 } } }
I20260926 03:31:48.187252 5823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83a4c45092c24c16a723f2ad0407b555 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83a4c45092c24c16a723f2ad0407b555" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a4c45092c24c16a723f2ad0407b555" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42045 } } }
I20260926 03:31:48.187798 5824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83a4c45092c24c16a723f2ad0407b555 [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:48.187906 5823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83a4c45092c24c16a723f2ad0407b555 [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:48.189427 5831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:31:48.195362 5831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:31:48.203483 5831 catalog_manager.cc:1347] Generated new cluster ID: a8a6f4e3a05b4e7c83c4ab9210df51b1
I20260926 03:31:48.203595 5831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:31:48.216641 5831 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:31:48.217895 5831 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:31:48.229444 5831 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83a4c45092c24c16a723f2ad0407b555: Generated new TSK 0
I20260926 03:31:48.230388 5831 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:31:48.257797 5820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:48.250381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48300 (local address 127.0.97.62:42045)
0926 03:31:48.250566 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:48.250573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:48.251082 (+ 509us) server_negotiation.cc:407] Connection header received
0926 03:31:48.251359 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:48.251366 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:48.251429 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:48.251574 (+ 145us) server_negotiation.cc:226] Negotiated authn=JWT
0926 03:31:48.252054 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:48.252813 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:48.254156 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:48.254625 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:48.254810 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:48.255300 (+ 490us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 03:31:48.257152 (+ 1852us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34627/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 03:31:48.257474 (+ 322us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48300: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34627/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 03:31:48.257967 5846 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:31:48.250318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42045 (local address 127.0.0.1:48300)
0926 03:31:48.250966 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:48.250980 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:48.251164 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:48.251610 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:48.251615 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:48.251629 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0926 03:31:48.251910 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:48.251919 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:48.252950 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:48.252956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:48.254000 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:48.254010 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:48.254807 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:48.254878 (+ 71us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:48.255013 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:48.255044 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 03:31:48.257336 (+ 2292us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34627/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 03:31:48.257576 (+ 240us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:42045: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34627/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":374,"threads_started":1}
W20260926 03:31:48.258350 5820 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48300: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34627/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260926 03:31:48.258971 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 5751
I20260926 03:31:48.260046 5812 generic_service.cc:196] Checking for leaks (request via RPC)
E20260926 03:31:48.269891 5847 webserver.cc:533] Webserver: sslize failed:
E20260926 03:31:48.269972 5847 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 03:31:48.350572 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 5751
2026-09-26T03:31:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (694 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T03:31:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:31:48Z Disabled control of system clock
I20260926 03:31:48.429565 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46083
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46083
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790393463219689-388-0/testchainca.cert with env {}
W20260926 03:31:48.743273 5852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:48.743537 5852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:48.743624 5852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:48.750612 5852 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:31:48.750711 5852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:48.750748 5852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:48.750797 5852 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:31:48.750841 5852 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 03:31:48.750893 5852 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 03:31:48.750939 5852 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 03:31:48.760406 5852 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46083
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790393463219689-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:48.761652 5852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:48.762830 5852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:48.774670 5860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:48.774796 5858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:48.774775 5857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:48.775314 5852 server_base.cc:1034] running on GCE node
I20260926 03:31:48.776108 5852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:48.777496 5852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:48.778728 5852 hybrid_clock.cc:648] HybridClock initialized: now 1790393508778687 us; error 54 us; skew 500 ppm
I20260926 03:31:48.779069 5852 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:31:48.781342 5852 webserver.cc:460] Webserver started at http://127.0.97.62:45097/ using document root <none> and password file <none>
I20260926 03:31:48.781994 5852 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:48.782126 5852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:48.782435 5852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:48.785238 5852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "e6999ac7eed0486abecb3d32a658e6eb"
format_stamp: "Formatted at 2026-09-26 03:31:48 on dist-test-slave-dt7p"
server_key: "167daf251d196d78833df2d218be973a"
server_key_iv: "c6b01cd7168c55da5882840c34446a40"
server_key_version: "encryptionkey@0"
I20260926 03:31:48.786011 5852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "e6999ac7eed0486abecb3d32a658e6eb"
format_stamp: "Formatted at 2026-09-26 03:31:48 on dist-test-slave-dt7p"
server_key: "167daf251d196d78833df2d218be973a"
server_key_iv: "c6b01cd7168c55da5882840c34446a40"
server_key_version: "encryptionkey@0"
I20260926 03:31:48.790961 5852 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:31:48.794179 5866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:48.795447 5852 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:31:48.795610 5852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "e6999ac7eed0486abecb3d32a658e6eb"
format_stamp: "Formatted at 2026-09-26 03:31:48 on dist-test-slave-dt7p"
server_key: "167daf251d196d78833df2d218be973a"
server_key_iv: "c6b01cd7168c55da5882840c34446a40"
server_key_version: "encryptionkey@0"
I20260926 03:31:48.795809 5852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:48.814806 5852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:48.815627 5852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:48.815893 5852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:48.833454 5852 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46083
I20260926 03:31:48.833477 5917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46083 every 8 connection(s)
I20260926 03:31:48.834724 5852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:31:48.836280 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 5852
I20260926 03:31:48.836494 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:31:48.837329 388 external_mini_cluster.cc:1428] Setting key 3c57850f37334752a917d8f83294bd10
I20260926 03:31:48.839160 5918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:48.849675 5921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:48.839139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35048 (local address 127.0.97.62:46083)
0926 03:31:48.840061 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:48.840071 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:48.840109 (+ 38us) server_negotiation.cc:407] Connection header received
0926 03:31:48.841638 (+ 1529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:48.841656 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:48.842070 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:48.842465 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:48.842984 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:48.844260 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:48.845604 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:48.846217 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:48.846409 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:48.847574 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:48.847580 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:48.847592 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:31:48.848101 (+ 509us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:48.848177 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:31:48.848195 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:48.848314 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:48.848558 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 03:31:48.848843 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":321,"threads_started":1}
I20260926 03:31:48.850194 5918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6999ac7eed0486abecb3d32a658e6eb: Bootstrap starting.
I20260926 03:31:48.853581 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:31:48.853700 5918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6999ac7eed0486abecb3d32a658e6eb: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:48.854832 5918 log.cc:824] T 00000000000000000000000000000000 P e6999ac7eed0486abecb3d32a658e6eb: Log is configured to *not* fsync() on all Append() calls
I20260926 03:31:48.857733 5918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6999ac7eed0486abecb3d32a658e6eb: No bootstrap required, opened a new log
I20260926 03:31:48.864830 5918 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6999ac7eed0486abecb3d32a658e6eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6999ac7eed0486abecb3d32a658e6eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } }
I20260926 03:31:48.865156 5918 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6999ac7eed0486abecb3d32a658e6eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:48.865229 5918 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6999ac7eed0486abecb3d32a658e6eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6999ac7eed0486abecb3d32a658e6eb, State: Initialized, Role: FOLLOWER
I20260926 03:31:48.865736 5918 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6999ac7eed0486abecb3d32a658e6eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6999ac7eed0486abecb3d32a658e6eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } }
I20260926 03:31:48.865947 5918 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6999ac7eed0486abecb3d32a658e6eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:31:48.866068 5918 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6999ac7eed0486abecb3d32a658e6eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:31:48.866187 5918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6999ac7eed0486abecb3d32a658e6eb [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:48.868312 5918 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6999ac7eed0486abecb3d32a658e6eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6999ac7eed0486abecb3d32a658e6eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } }
I20260926 03:31:48.868803 5918 leader_election.cc:302] T 00000000000000000000000000000000 P e6999ac7eed0486abecb3d32a658e6eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6999ac7eed0486abecb3d32a658e6eb; no voters:
I20260926 03:31:48.869292 5918 leader_election.cc:288] T 00000000000000000000000000000000 P e6999ac7eed0486abecb3d32a658e6eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:31:48.869443 5923 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6999ac7eed0486abecb3d32a658e6eb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:31:48.870967 5923 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6999ac7eed0486abecb3d32a658e6eb [term 1 LEADER]: Becoming Leader. State: Replica: e6999ac7eed0486abecb3d32a658e6eb, State: Running, Role: LEADER
I20260926 03:31:48.871320 5918 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6999ac7eed0486abecb3d32a658e6eb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:31:48.871481 5923 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6999ac7eed0486abecb3d32a658e6eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6999ac7eed0486abecb3d32a658e6eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } }
I20260926 03:31:48.874185 5925 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6999ac7eed0486abecb3d32a658e6eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6999ac7eed0486abecb3d32a658e6eb. Latest consensus state: current_term: 1 leader_uuid: "e6999ac7eed0486abecb3d32a658e6eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6999ac7eed0486abecb3d32a658e6eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } }
I20260926 03:31:48.874158 5924 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6999ac7eed0486abecb3d32a658e6eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6999ac7eed0486abecb3d32a658e6eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6999ac7eed0486abecb3d32a658e6eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } }
I20260926 03:31:48.874576 5924 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6999ac7eed0486abecb3d32a658e6eb [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:48.874580 5925 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6999ac7eed0486abecb3d32a658e6eb [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:48.876372 5930 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:31:48.881652 5930 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:31:48.889639 5930 catalog_manager.cc:1347] Generated new cluster ID: faa6ad9d181b4071ae8d14ff68e4d384
I20260926 03:31:48.889739 5930 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:31:48.903535 5930 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:31:48.905158 5930 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:31:48.914533 5930 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6999ac7eed0486abecb3d32a658e6eb: Generated new TSK 0
I20260926 03:31:48.915552 5930 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:31:48.981628 5921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:48.928664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35064 (local address 127.0.97.62:46083)
0926 03:31:48.928813 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:48.928818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:48.929290 (+ 472us) server_negotiation.cc:407] Connection header received
0926 03:31:48.929539 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:48.929543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:48.929610 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:48.929766 (+ 156us) server_negotiation.cc:226] Negotiated authn=JWT
0926 03:31:48.930232 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:48.930943 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:48.932316 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:48.932824 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:48.932958 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:48.933423 (+ 465us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 03:31:48.980827 (+ 47404us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 03:31:48.981303 (+ 476us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35064: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 03:31:48.981786 5921 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35064: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 03:31:48.981842 5947 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:31:48.928653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.0.1:35064)
0926 03:31:48.929189 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:48.929204 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:48.929367 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:48.929774 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:48.929778 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:48.929790 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0926 03:31:48.930080 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:48.930088 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:48.931090 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:48.931097 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:48.932176 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:48.932186 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:48.932943 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:48.932991 (+ 48us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:48.933142 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:48.933173 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 03:31:48.981146 (+ 47973us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 03:31:48.981425 (+ 279us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:46083: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":290,"threads_started":1}
I20260926 03:31:48.983004 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 5852
I20260926 03:31:48.984110 5913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:31:49.079887 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 5852
2026-09-26T03:31:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (708 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (404 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (402 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:31:49 dist-test-slave-dt7p krb5kdc[5962](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:31:49 dist-test-slave-dt7p krb5kdc[5962](info): set up 2 sockets
Sep 26 03:31:49 dist-test-slave-dt7p krb5kdc[5962](info): commencing operation
krb5kdc: starting...
W20260926 03:31:50.969748 388 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:31:51 dist-test-slave-dt7p krb5kdc[5962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:31:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:31:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:31:51.173218 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44527
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44527
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:51.486399 5975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:51.486713 5975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:51.486896 5975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:51.495471 5975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:31:51.495587 5975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:51.495700 5975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:51.495819 5975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:31:51.495909 5975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:31:51.506589 5975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:51.507899 5975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:51.509191 5975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:51.525151 5980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:51.525174 5983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:51.525085 5981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:51.525460 5975 server_base.cc:1034] running on GCE node
I20260926 03:31:51.526556 5975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:51.527947 5975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:51.529171 5975 hybrid_clock.cc:648] HybridClock initialized: now 1790393511529127 us; error 35 us; skew 500 ppm
I20260926 03:31:51.529525 5975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:51 dist-test-slave-dt7p krb5kdc[5962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:51.532379 5975 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:31:51.533759 5975 webserver.cc:460] Webserver started at http://127.0.97.62:35839/ using document root <none> and password file <none>
I20260926 03:31:51.534400 5975 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:51.534543 5975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:51.534840 5975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:51.537837 5975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "091418a81b59439aa747fab32f5dd10f"
format_stamp: "Formatted at 2026-09-26 03:31:51 on dist-test-slave-dt7p"
server_key: "632e3eac6273eff9d3e7381c06614409"
server_key_iv: "b8c98d90d1127ab5fd36becda1f443e3"
server_key_version: "encryptionkey@0"
I20260926 03:31:51.538589 5975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "091418a81b59439aa747fab32f5dd10f"
format_stamp: "Formatted at 2026-09-26 03:31:51 on dist-test-slave-dt7p"
server_key: "632e3eac6273eff9d3e7381c06614409"
server_key_iv: "b8c98d90d1127ab5fd36becda1f443e3"
server_key_version: "encryptionkey@0"
I20260926 03:31:51.543371 5975 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 03:31:51.546670 5990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:51.548010 5975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:31:51.548197 5975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "091418a81b59439aa747fab32f5dd10f"
format_stamp: "Formatted at 2026-09-26 03:31:51 on dist-test-slave-dt7p"
server_key: "632e3eac6273eff9d3e7381c06614409"
server_key_iv: "b8c98d90d1127ab5fd36becda1f443e3"
server_key_version: "encryptionkey@0"
I20260926 03:31:51.548385 5975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:51.566811 5975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:51.570693 5975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:51.570966 5975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:51.588650 5975 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44527
I20260926 03:31:51.588732 6041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44527 every 8 connection(s)
I20260926 03:31:51.589905 5975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:31:51.591548 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 5975
I20260926 03:31:51.591768 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:31:51.592634 388 external_mini_cluster.cc:1428] Setting key 490414864859c5d3f9cd12362c4b6e23
I20260926 03:31:51.594264 6042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 03:31:51 dist-test-slave-dt7p krb5kdc[5962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:31:51.607692 6042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 091418a81b59439aa747fab32f5dd10f: Bootstrap starting.
I20260926 03:31:51.611070 6042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 091418a81b59439aa747fab32f5dd10f: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:51.612069 6042 log.cc:824] T 00000000000000000000000000000000 P 091418a81b59439aa747fab32f5dd10f: Log is configured to *not* fsync() on all Append() calls
I20260926 03:31:51.614842 6042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 091418a81b59439aa747fab32f5dd10f: No bootstrap required, opened a new log
I20260926 03:31:51.614868 6045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:51.594634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55740 (local address 127.0.97.62:44527)
0926 03:31:51.595361 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:51.595372 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:51.595411 (+ 39us) server_negotiation.cc:407] Connection header received
0926 03:31:51.597041 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:51.597060 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:51.597407 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:51.597713 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:51.599403 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:51.600828 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:51.602559 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:51.603172 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:51.603293 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:51.607918 (+ 4625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:51.607926 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:51.607940 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:51.607974 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:51.610910 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:51.611671 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:51.611679 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:51.611691 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:51.611857 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:51.612223 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:51.612229 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:51.612232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:51.612818 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:51.613124 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:51.613540 (+ 416us) server_negotiation.cc:299] Negotiation successful
0926 03:31:51.613949 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":357,"threads_started":1}
I20260926 03:31:51.622700 6042 raft_consensus.cc:348] T 00000000000000000000000000000000 P 091418a81b59439aa747fab32f5dd10f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "091418a81b59439aa747fab32f5dd10f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44527 } }
I20260926 03:31:51.623047 6042 raft_consensus.cc:374] T 00000000000000000000000000000000 P 091418a81b59439aa747fab32f5dd10f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:51.623157 6042 raft_consensus.cc:729] T 00000000000000000000000000000000 P 091418a81b59439aa747fab32f5dd10f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 091418a81b59439aa747fab32f5dd10f, State: Initialized, Role: FOLLOWER
I20260926 03:31:51.623728 6042 consensus_queue.cc:260] T 00000000000000000000000000000000 P 091418a81b59439aa747fab32f5dd10f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "091418a81b59439aa747fab32f5dd10f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44527 } }
I20260926 03:31:51.623950 6042 raft_consensus.cc:388] T 00000000000000000000000000000000 P 091418a81b59439aa747fab32f5dd10f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:31:51.624085 6042 raft_consensus.cc:482] T 00000000000000000000000000000000 P 091418a81b59439aa747fab32f5dd10f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:31:51.624266 6042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 091418a81b59439aa747fab32f5dd10f [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:51.626590 6042 raft_consensus.cc:504] T 00000000000000000000000000000000 P 091418a81b59439aa747fab32f5dd10f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "091418a81b59439aa747fab32f5dd10f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44527 } }
I20260926 03:31:51.627050 6042 leader_election.cc:302] T 00000000000000000000000000000000 P 091418a81b59439aa747fab32f5dd10f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 091418a81b59439aa747fab32f5dd10f; no voters:
I20260926 03:31:51.627579 6042 leader_election.cc:288] T 00000000000000000000000000000000 P 091418a81b59439aa747fab32f5dd10f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:31:51.627701 6047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 091418a81b59439aa747fab32f5dd10f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:31:51.628343 6047 raft_consensus.cc:686] T 00000000000000000000000000000000 P 091418a81b59439aa747fab32f5dd10f [term 1 LEADER]: Becoming Leader. State: Replica: 091418a81b59439aa747fab32f5dd10f, State: Running, Role: LEADER
I20260926 03:31:51.628842 6047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 091418a81b59439aa747fab32f5dd10f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "091418a81b59439aa747fab32f5dd10f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44527 } }
I20260926 03:31:51.629550 6042 sys_catalog.cc:564] T 00000000000000000000000000000000 P 091418a81b59439aa747fab32f5dd10f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:31:51.631510 6048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 091418a81b59439aa747fab32f5dd10f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "091418a81b59439aa747fab32f5dd10f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "091418a81b59439aa747fab32f5dd10f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44527 } } }
I20260926 03:31:51.631851 6048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 091418a81b59439aa747fab32f5dd10f [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:51.636888 6049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 091418a81b59439aa747fab32f5dd10f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 091418a81b59439aa747fab32f5dd10f. Latest consensus state: current_term: 1 leader_uuid: "091418a81b59439aa747fab32f5dd10f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "091418a81b59439aa747fab32f5dd10f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44527 } } }
I20260926 03:31:51.637606 6054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:31:51.637625 6049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 091418a81b59439aa747fab32f5dd10f [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:51.643711 6054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:31:51.652294 6054 catalog_manager.cc:1347] Generated new cluster ID: 299051a3e5014253a287fd8c44df424b
I20260926 03:31:51.652406 6054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:31:51.669754 6054 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:31:51.671015 6054 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:31:51.679910 6054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 091418a81b59439aa747fab32f5dd10f: Generated new TSK 0
I20260926 03:31:51.680860 6054 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:31:51.735497 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44527
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:52.059094 6068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:52.059366 6068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:52.059538 6068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:52.066635 6068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:52.066741 6068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:52.066968 6068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:31:52.076681 6068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:52.077931 6068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:52.079195 6068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:52.092604 6076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:52.092631 6074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:52.092911 6073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:52.093286 6068 server_base.cc:1034] running on GCE node
I20260926 03:31:52.093901 6068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:52.094842 6068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:52.096087 6068 hybrid_clock.cc:648] HybridClock initialized: now 1790393512096023 us; error 70 us; skew 500 ppm
I20260926 03:31:52.096451 6068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:52 dist-test-slave-dt7p krb5kdc[5962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:52.106243 6068 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:31:52.107868 6068 webserver.cc:460] Webserver started at http://127.0.97.1:46361/ using document root <none> and password file <none>
I20260926 03:31:52.108552 6068 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:52.108740 6068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:52.109055 6068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:52.111989 6068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-0/data/instance:
uuid: "fec203be10db41e190098aa3f81d3020"
format_stamp: "Formatted at 2026-09-26 03:31:52 on dist-test-slave-dt7p"
server_key: "d9721852f763306f6c28a16f566f80b6"
server_key_iv: "77e28fe51589d7b5eecd3826482ec0f0"
server_key_version: "encryptionkey@0"
I20260926 03:31:52.112831 6068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-0/wal/instance:
uuid: "fec203be10db41e190098aa3f81d3020"
format_stamp: "Formatted at 2026-09-26 03:31:52 on dist-test-slave-dt7p"
server_key: "d9721852f763306f6c28a16f566f80b6"
server_key_iv: "77e28fe51589d7b5eecd3826482ec0f0"
server_key_version: "encryptionkey@0"
I20260926 03:31:52.117916 6068 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 03:31:52.121366 6083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:52.122735 6068 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:31:52.122924 6068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-0/wal
uuid: "fec203be10db41e190098aa3f81d3020"
format_stamp: "Formatted at 2026-09-26 03:31:52 on dist-test-slave-dt7p"
server_key: "d9721852f763306f6c28a16f566f80b6"
server_key_iv: "77e28fe51589d7b5eecd3826482ec0f0"
server_key_version: "encryptionkey@0"
I20260926 03:31:52.123142 6068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:52.165652 6068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:52.171384 6068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:52.171741 6068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:52.173022 6068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:31:52.174899 6068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:31:52.175014 6068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:52.175134 6068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:31:52.175196 6068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:52.211592 6068 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46387
I20260926 03:31:52.211639 6195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46387 every 8 connection(s)
I20260926 03:31:52.212915 6068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:31:52.219767 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 6068
I20260926 03:31:52.219983 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:31:52.220706 388 external_mini_cluster.cc:1428] Setting key f3583278dd491a4546028b457c45aa9c
Sep 26 03:31:52 dist-test-slave-dt7p krb5kdc[5962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:31:52.234043 6198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:52.215618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52121 (local address 127.0.97.62:44527)
0926 03:31:52.216075 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:52.216079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:52.217045 (+ 966us) server_negotiation.cc:407] Connection header received
0926 03:31:52.218800 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:52.218805 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:52.218897 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:52.219024 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:52.220601 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:52.221571 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:52.223239 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:52.223629 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:52.223774 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:52.227591 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:52.227600 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:52.227608 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:52.227654 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:52.230682 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:52.231636 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:52.231640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:52.231642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:52.231752 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:52.232254 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:52.232259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:52.232261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:52.232545 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:52.232756 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:52.233619 (+ 863us) server_negotiation.cc:299] Negotiation successful
0926 03:31:52.233781 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":254,"threads_started":1}
I20260926 03:31:52.235601 6199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:52.216146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44527 (local address 127.0.97.1:52121)
0926 03:31:52.216875 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:52.216926 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:52.218516 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:52.219253 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:52.219260 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:52.219745 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:52.220408 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:52.220426 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:52.221753 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:52.221757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:52.223083 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:52.223095 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:52.223856 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:52.223865 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:52.224095 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:52.225004 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:52.225026 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:52.227352 (+ 2326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:52.230838 (+ 3486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:52.230850 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:52.230869 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:52.231511 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:52.231860 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:52.231867 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:52.231874 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:52.232110 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:52.232759 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:52.232770 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:52.233235 (+ 465us) client_negotiation.cc:769] Sending connection context
0926 03:31:52.233533 (+ 298us) client_negotiation.cc:240] Negotiation successful
0926 03:31:52.234116 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":288,"threads_started":1}
I20260926 03:31:52.237251 6196 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44527
I20260926 03:31:52.237556 6196 heartbeater.cc:464] Registering TS with master...
I20260926 03:31:52.238448 6196 heartbeater.cc:511] Master 127.0.97.62:44527 requested a full tablet report, sending...
I20260926 03:31:52.240643 6007 ts_manager.cc:194] Registered new tserver with Master: fec203be10db41e190098aa3f81d3020 (127.0.97.1:46387)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 03:31:52.242142 6007 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52121
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:31:52.267786 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44527
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:52.588064 6202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:52.588460 6202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:52.588668 6202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:52.596115 6202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:52.596266 6202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:52.596511 6202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:31:52.606406 6202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:52.607575 6202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:52.608834 6202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:52.621362 6207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:52.621330 6208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:52.621338 6210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:52.622256 6202 server_base.cc:1034] running on GCE node
I20260926 03:31:52.622921 6202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:52.623772 6202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:52.625000 6202 hybrid_clock.cc:648] HybridClock initialized: now 1790393512624953 us; error 61 us; skew 500 ppm
I20260926 03:31:52.625311 6202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:52 dist-test-slave-dt7p krb5kdc[5962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:52.628536 6202 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:31:52.630059 6202 webserver.cc:460] Webserver started at http://127.0.97.2:46715/ using document root <none> and password file <none>
I20260926 03:31:52.630808 6202 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:52.630963 6202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:52.631280 6202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:52.634567 6202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-1/data/instance:
uuid: "ec91a740d44f4db78bcb48c02427c675"
format_stamp: "Formatted at 2026-09-26 03:31:52 on dist-test-slave-dt7p"
server_key: "ae69e8c53545746af729e0a336cece90"
server_key_iv: "affbb0757aa856c15e3c725318bd7433"
server_key_version: "encryptionkey@0"
I20260926 03:31:52.635399 6202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ec91a740d44f4db78bcb48c02427c675"
format_stamp: "Formatted at 2026-09-26 03:31:52 on dist-test-slave-dt7p"
server_key: "ae69e8c53545746af729e0a336cece90"
server_key_iv: "affbb0757aa856c15e3c725318bd7433"
server_key_version: "encryptionkey@0"
I20260926 03:31:52.640796 6202 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 03:31:52.644177 6217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:52.645615 6202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:31:52.645798 6202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-1/wal
uuid: "ec91a740d44f4db78bcb48c02427c675"
format_stamp: "Formatted at 2026-09-26 03:31:52 on dist-test-slave-dt7p"
server_key: "ae69e8c53545746af729e0a336cece90"
server_key_iv: "affbb0757aa856c15e3c725318bd7433"
server_key_version: "encryptionkey@0"
I20260926 03:31:52.646013 6202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:52.660511 6202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:52.664479 6202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:52.664806 6202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:52.665808 6202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:31:52.667089 6202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:31:52.667208 6202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:52.667358 6202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:31:52.667438 6202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:52.701135 6202 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44631
I20260926 03:31:52.701157 6329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44631 every 8 connection(s)
I20260926 03:31:52.702416 6202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:31:52.706987 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 6202
I20260926 03:31:52.707195 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:31:52.707721 388 external_mini_cluster.cc:1428] Setting key 8443c2ef1f6f5e40dd03ca891ce4e4ba
Sep 26 03:31:52 dist-test-slave-dt7p krb5kdc[5962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:31:52.724043 6198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:52.705293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50231 (local address 127.0.97.62:44527)
0926 03:31:52.705520 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:52.705527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:52.706807 (+ 1280us) server_negotiation.cc:407] Connection header received
0926 03:31:52.709118 (+ 2311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:52.709123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:52.709208 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:52.709327 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:52.711238 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:52.711935 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:52.714362 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:52.714757 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:52.714906 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:52.718498 (+ 3592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:52.718503 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:52.718506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:52.718536 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:52.721047 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:52.721792 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:52.721798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:52.721802 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:52.721927 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:52.722386 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:52.722394 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:52.722397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:52.722654 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:52.722810 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:52.723468 (+ 658us) server_negotiation.cc:299] Negotiation successful
0926 03:31:52.723687 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 03:31:52.725114 6332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:52.705791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44527 (local address 127.0.97.2:50231)
0926 03:31:52.706610 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:52.706659 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:52.708862 (+ 2203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:52.709549 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:52.709557 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:52.710266 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:52.711058 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:52.711084 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:52.712089 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:52.712097 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:52.714211 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:52.714229 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:52.714991 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:52.714998 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:52.715222 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:52.715925 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:52.715946 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:52.718269 (+ 2323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:52.721210 (+ 2941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:52.721218 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:52.721234 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:52.721642 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:52.722043 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:52.722050 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:52.722053 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:52.722260 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:52.722783 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:52.722790 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:52.723124 (+ 334us) client_negotiation.cc:769] Sending connection context
0926 03:31:52.723382 (+ 258us) client_negotiation.cc:240] Negotiation successful
0926 03:31:52.723918 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":319,"threads_started":1}
I20260926 03:31:52.726656 6330 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44527
I20260926 03:31:52.726960 6330 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 03:31:52.727859 6330 heartbeater.cc:511] Master 127.0.97.62:44527 requested a full tablet report, sending...
I20260926 03:31:52.729336 6007 ts_manager.cc:194] Registered new tserver with Master: ec91a740d44f4db78bcb48c02427c675 (127.0.97.2:44631)
I20260926 03:31:52.730266 6007 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50231
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:31:52.756215 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44527
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:53.069769 6335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:53.070080 6335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:53.070247 6335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:53.077366 6335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:53.077519 6335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:53.077692 6335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:31:53.087016 6335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:53.088253 6335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:53.089493 6335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:53.102190 6340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:53.102087 6341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:53.102114 6343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:53.103374 6335 server_base.cc:1034] running on GCE node
I20260926 03:31:53.103932 6335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:53.104946 6335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:53.106173 6335 hybrid_clock.cc:648] HybridClock initialized: now 1790393513106136 us; error 67 us; skew 500 ppm
I20260926 03:31:53.106478 6335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:53 dist-test-slave-dt7p krb5kdc[5962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:53.109714 6335 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:31:53.111112 6335 webserver.cc:460] Webserver started at http://127.0.97.3:39561/ using document root <none> and password file <none>
I20260926 03:31:53.111795 6335 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:53.111946 6335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:53.112247 6335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:53.115159 6335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-2/data/instance:
uuid: "d401f42032064a2484c9a3fc28b6bd2e"
format_stamp: "Formatted at 2026-09-26 03:31:53 on dist-test-slave-dt7p"
server_key: "58c4ec4d630e72235c77eb3271b692a6"
server_key_iv: "383ffaa97c7ef9a10a585cef5d397481"
server_key_version: "encryptionkey@0"
I20260926 03:31:53.115916 6335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d401f42032064a2484c9a3fc28b6bd2e"
format_stamp: "Formatted at 2026-09-26 03:31:53 on dist-test-slave-dt7p"
server_key: "58c4ec4d630e72235c77eb3271b692a6"
server_key_iv: "383ffaa97c7ef9a10a585cef5d397481"
server_key_version: "encryptionkey@0"
I20260926 03:31:53.120932 6335 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 03:31:53.124291 6350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:53.125744 6335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:31:53.125928 6335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-2/wal
uuid: "d401f42032064a2484c9a3fc28b6bd2e"
format_stamp: "Formatted at 2026-09-26 03:31:53 on dist-test-slave-dt7p"
server_key: "58c4ec4d630e72235c77eb3271b692a6"
server_key_iv: "383ffaa97c7ef9a10a585cef5d397481"
server_key_version: "encryptionkey@0"
I20260926 03:31:53.126137 6335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:53.152760 6335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:53.156781 6335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:53.157080 6335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:53.158061 6335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:31:53.159397 6335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:31:53.159525 6335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:53.159648 6335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:31:53.159716 6335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:53.192698 6335 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33281
I20260926 03:31:53.192843 6462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33281 every 8 connection(s)
I20260926 03:31:53.194048 6335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:31:53.204449 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 6335
I20260926 03:31:53.204722 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:31:53.205183 388 external_mini_cluster.cc:1428] Setting key 72eec66749245809765dc1185b9cb88c
Sep 26 03:31:53 dist-test-slave-dt7p krb5kdc[5962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:31:53.213723 6198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:53.196740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46103 (local address 127.0.97.62:44527)
0926 03:31:53.196920 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:53.196929 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:53.198157 (+ 1228us) server_negotiation.cc:407] Connection header received
0926 03:31:53.199916 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:53.199921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:53.200032 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:53.200152 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:53.201666 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:53.202406 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:53.203922 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:53.204409 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:53.204553 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:53.208227 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:53.208232 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:53.208236 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:53.208266 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:53.210701 (+ 2435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:53.211410 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:53.211416 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:53.211419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:53.211547 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:53.211982 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:53.211987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:53.211990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:53.212253 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:53.212415 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:53.213163 (+ 748us) server_negotiation.cc:299] Negotiation successful
0926 03:31:53.213375 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 03:31:53.214629 6465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:53.197198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44527 (local address 127.0.97.3:46103)
0926 03:31:53.197999 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:53.198037 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:53.199640 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:53.200375 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:53.200384 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:53.200953 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:53.201477 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:53.201493 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:53.202573 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:53.202582 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:53.203764 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:53.203779 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:53.204658 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:53.204666 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:53.204970 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:53.205677 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:53.205700 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:53.208015 (+ 2315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:53.210847 (+ 2832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:53.210855 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:53.210872 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:53.211278 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:53.211662 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:53.211668 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:53.211672 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:53.211856 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:53.212387 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:53.212395 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:53.212780 (+ 385us) client_negotiation.cc:769] Sending connection context
0926 03:31:53.213044 (+ 264us) client_negotiation.cc:240] Negotiation successful
0926 03:31:53.213551 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":344,"threads_started":1}
I20260926 03:31:53.216085 6463 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44527
I20260926 03:31:53.216379 6463 heartbeater.cc:464] Registering TS with master...
I20260926 03:31:53.217109 6463 heartbeater.cc:511] Master 127.0.97.62:44527 requested a full tablet report, sending...
I20260926 03:31:53.218498 6007 ts_manager.cc:194] Registered new tserver with Master: d401f42032064a2484c9a3fc28b6bd2e (127.0.97.3:33281)
I20260926 03:31:53.219686 6007 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46103
I20260926 03:31:53.220491 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:31:53.245059 6196 heartbeater.cc:503] Master 127.0.97.62:44527 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/krb5kdc/test-admin.keytab.
Sep 26 03:31:53 dist-test-slave-dt7p krb5kdc[5962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:53.250774 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 03:31:53.255626 6470 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:31:53.253403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56304 (local address 127.0.97.1:46387)
0926 03:31:53.254079 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:53.254086 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:53.254114 (+ 28us) server_negotiation.cc:407] Connection header received
0926 03:31:53.254178 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:53.254185 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:53.254374 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:53.254501 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:53.255251 (+ 750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56304: BlockingRecv error: recv got EOF from 127.0.0.1:56304 (error 108)
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":359,"threads_started":1}
W20260926 03:31:53.255681 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/krb5kdc/krb5cc' of type FILE
Sep 26 03:31:53 dist-test-slave-dt7p krb5kdc[5962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:53.257403 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 03:31:53 dist-test-slave-dt7p krb5kdc[5962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:53.258750 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 03:31:53 dist-test-slave-dt7p krb5kdc[5962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:31:53.273180 6470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:53.260342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56312 (local address 127.0.97.1:46387)
0926 03:31:53.260533 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:53.260538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:53.260782 (+ 244us) server_negotiation.cc:407] Connection header received
0926 03:31:53.260903 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:53.260907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:53.260959 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:53.261034 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:53.261938 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:53.262955 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:53.264171 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:53.264786 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:53.264883 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:53.268191 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:53.268201 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:53.268214 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:53.268248 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:53.270855 (+ 2607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:53.271417 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:53.271424 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:53.271426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:53.271543 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:53.271850 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:53.271853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:53.271855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:53.272191 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:53.272385 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:53.272726 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 03:31:53.272935 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 03:31:53.278764 6470 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:31:53.277096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56316 (local address 127.0.97.1:46387)
0926 03:31:53.277257 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:53.277261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:53.277597 (+ 336us) server_negotiation.cc:407] Connection header received
0926 03:31:53.277713 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:53.277716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:53.277768 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:53.277842 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:53.278484 (+ 642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56316: BlockingRecv error: recv got EOF from 127.0.0.1:56316 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260926 03:31:53.278939 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/krb5kdc/krb5cc' of type FILE
Sep 26 03:31:53 dist-test-slave-dt7p krb5kdc[5962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:53.280416 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 03:31:53 dist-test-slave-dt7p krb5kdc[5962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:53.281776 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 03:31:53 dist-test-slave-dt7p krb5kdc[5962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:31:53.294449 6470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:53.283023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56318 (local address 127.0.97.1:46387)
0926 03:31:53.283235 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:53.283241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:53.283447 (+ 206us) server_negotiation.cc:407] Connection header received
0926 03:31:53.283558 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:53.283561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:53.283620 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:53.283694 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:53.284513 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:53.285232 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:53.286406 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:53.286834 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:53.286922 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:53.290179 (+ 3257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:53.290183 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:53.290187 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:53.290213 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:53.292459 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:53.293078 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:53.293082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:53.293084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:53.293189 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:53.293487 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:53.293490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:53.293492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:53.293717 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:53.293889 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:53.294109 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 03:31:53.294249 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260926 03:31:53.298802 6470 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:31:53.297049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56334 (local address 127.0.97.1:46387)
0926 03:31:53.297265 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:53.297271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:53.297491 (+ 220us) server_negotiation.cc:407] Connection header received
0926 03:31:53.297681 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:53.297686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:53.297781 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:53.297873 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:53.298584 (+ 711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56334: BlockingRecv error: recv got EOF from 127.0.0.1:56334 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 03:31:53.299079 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790393463219689-388-0/krb5kdc/krb5cc' of type FILE
Sep 26 03:31:53 dist-test-slave-dt7p krb5kdc[5962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:53.300657 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 03:31:53 dist-test-slave-dt7p krb5kdc[5962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:53.301975 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 03:31:53 dist-test-slave-dt7p krb5kdc[5962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:31:53.315778 6470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:53.303243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56342 (local address 127.0.97.1:46387)
0926 03:31:53.303427 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:53.303435 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:53.303712 (+ 277us) server_negotiation.cc:407] Connection header received
0926 03:31:53.303899 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:53.303903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:53.303955 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:53.304065 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:53.305113 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:53.305808 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:53.307042 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:53.307439 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:53.307575 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:53.310911 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:53.310916 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:53.310919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:53.310948 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:53.313364 (+ 2416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:53.313999 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:53.314006 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:53.314009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:53.314140 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:53.314499 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:53.314506 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:53.314509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:53.314767 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:53.315097 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:53.315283 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 03:31:53.315478 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 03:31:53.317250 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 6068
I20260926 03:31:53.329183 6470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:53.318026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56356 (local address 127.0.97.1:46387)
0926 03:31:53.318219 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:53.318225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:53.318447 (+ 222us) server_negotiation.cc:407] Connection header received
0926 03:31:53.318625 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:53.318630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:53.318687 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:53.318785 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:53.319669 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:53.320315 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:53.321587 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:53.321993 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:53.322132 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:53.324060 (+ 1928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:53.324066 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:53.324076 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:53.324103 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:53.326596 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:53.327295 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:53.327302 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:53.327306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:53.327427 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:53.327790 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:53.327796 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:53.327800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:53.328041 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:53.328337 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:53.328660 (+ 323us) server_negotiation.cc:299] Negotiation successful
0926 03:31:53.328855 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 03:31:53.329773 6191 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:31:53.411571 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 6068
I20260926 03:31:53.427580 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 6202
Sep 26 03:31:53 dist-test-slave-dt7p krb5kdc[5962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:31:53.443126 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:53.428472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36704 (local address 127.0.97.2:44631)
0926 03:31:53.429133 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:53.429140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:53.429166 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:31:53.429224 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:53.429230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:53.429451 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:53.429638 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:53.430687 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:53.431890 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:53.433124 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:53.433662 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:53.433765 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:53.437427 (+ 3662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:53.437438 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:53.437452 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:53.437483 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:53.440463 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:53.441161 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:53.441168 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:53.441171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:53.441283 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:53.441623 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:53.441626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:53.441628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:53.441957 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:53.442218 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:53.442626 (+ 408us) server_negotiation.cc:299] Negotiation successful
0926 03:31:53.442842 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":375,"threads_started":1}
I20260926 03:31:53.444389 6325 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:31:53.526028 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 6202
I20260926 03:31:53.541842 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 6335
Sep 26 03:31:53 dist-test-slave-dt7p krb5kdc[5962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:31:53.557999 6485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:53.542866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53232 (local address 127.0.97.3:33281)
0926 03:31:53.543509 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:53.543516 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:53.543546 (+ 30us) server_negotiation.cc:407] Connection header received
0926 03:31:53.543607 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:53.543614 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:53.543827 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:53.543950 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:53.545011 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:53.546006 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:53.547160 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:53.547700 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:53.547806 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:53.551600 (+ 3794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:53.551612 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:53.551626 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:53.551660 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:53.554780 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:53.555972 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:53.555979 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:53.555981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:53.556103 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:53.556450 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:53.556453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:53.556455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:53.556891 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:53.557167 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:53.557461 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 03:31:53.557672 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":345,"threads_started":1}
I20260926 03:31:53.559317 6458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:31:53.641680 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 6335
I20260926 03:31:53.657716 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 5975
I20260926 03:31:53.658954 6037 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:31:53.764427 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 5975
2026-09-26T03:31:53Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3883 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:31:53 dist-test-slave-dt7p krb5kdc[6490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:31:53 dist-test-slave-dt7p krb5kdc[6490](info): set up 2 sockets
Sep 26 03:31:53 dist-test-slave-dt7p krb5kdc[6490](info): commencing operation
krb5kdc: starting...
W20260926 03:31:54.865331 388 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:31:55 dist-test-slave-dt7p krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:31:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:31:55Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260926 03:31:55.077116 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46591
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46591
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:55.399091 6503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:55.399405 6503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:55.399492 6503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:55.407138 6503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:31:55.407256 6503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:55.407316 6503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:55.407368 6503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:31:55.407413 6503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:31:55.417287 6503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:55.418587 6503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:55.419778 6503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:55.431176 6509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:55.431286 6508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:55.431246 6511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:55.431951 6503 server_base.cc:1034] running on GCE node
I20260926 03:31:55.432880 6503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:55.434250 6503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:55.435526 6503 hybrid_clock.cc:648] HybridClock initialized: now 1790393515435440 us; error 43 us; skew 500 ppm
I20260926 03:31:55.435930 6503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:55 dist-test-slave-dt7p krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393515, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:55.439067 6503 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260926 03:31:55.440415 6503 webserver.cc:460] Webserver started at http://127.0.97.62:46543/ using document root <none> and password file <none>
I20260926 03:31:55.441125 6503 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:55.441222 6503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:55.441461 6503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:55.444301 6503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "aaf83112ebc946f5b71d630f6d3454b0"
format_stamp: "Formatted at 2026-09-26 03:31:55 on dist-test-slave-dt7p"
server_key: "d21a20860df9fea13bd318c1545ca5d1"
server_key_iv: "ac74369346a5206c3be44dbcd79fabaf"
server_key_version: "encryptionkey@0"
I20260926 03:31:55.445096 6503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "aaf83112ebc946f5b71d630f6d3454b0"
format_stamp: "Formatted at 2026-09-26 03:31:55 on dist-test-slave-dt7p"
server_key: "d21a20860df9fea13bd318c1545ca5d1"
server_key_iv: "ac74369346a5206c3be44dbcd79fabaf"
server_key_version: "encryptionkey@0"
I20260926 03:31:55.449936 6503 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 03:31:55.453135 6518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:55.454435 6503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:31:55.454625 6503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "aaf83112ebc946f5b71d630f6d3454b0"
format_stamp: "Formatted at 2026-09-26 03:31:55 on dist-test-slave-dt7p"
server_key: "d21a20860df9fea13bd318c1545ca5d1"
server_key_iv: "ac74369346a5206c3be44dbcd79fabaf"
server_key_version: "encryptionkey@0"
I20260926 03:31:55.454824 6503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:55.469074 6503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:55.472790 6503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:55.473073 6503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:55.491446 6503 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46591
I20260926 03:31:55.491477 6569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46591 every 8 connection(s)
I20260926 03:31:55.492835 6503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:31:55.496536 6570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:55.498023 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 6503
I20260926 03:31:55.498263 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:31:55.499035 388 external_mini_cluster.cc:1428] Setting key f8300aac27d3d48b11f932eb7e768ffb
I20260926 03:31:55.510970 6570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0: Bootstrap starting.
Sep 26 03:31:55 dist-test-slave-dt7p krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:31:55.514638 6570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:55.515657 6570 log.cc:824] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0: Log is configured to *not* fsync() on all Append() calls
I20260926 03:31:55.518678 6570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0: No bootstrap required, opened a new log
I20260926 03:31:55.522023 6573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:55.501073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54288 (local address 127.0.97.62:46591)
0926 03:31:55.501872 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:55.501882 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:55.501925 (+ 43us) server_negotiation.cc:407] Connection header received
0926 03:31:55.503571 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:55.503604 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:55.504116 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:55.504429 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:55.506254 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:55.507583 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:55.509356 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:55.510017 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:55.510146 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:55.514684 (+ 4538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:55.514691 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:55.514704 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:55.514741 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:55.518154 (+ 3413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:55.518848 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:55.518861 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:55.518867 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:55.519024 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:55.519411 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:55.519416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:55.519424 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:55.519878 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:55.520196 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:55.520671 (+ 475us) server_negotiation.cc:299] Negotiation successful
0926 03:31:55.521051 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":397,"threads_started":1}
I20260926 03:31:55.527047 6570 raft_consensus.cc:348] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } }
I20260926 03:31:55.527428 6570 raft_consensus.cc:374] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:55.527540 6570 raft_consensus.cc:729] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf83112ebc946f5b71d630f6d3454b0, State: Initialized, Role: FOLLOWER
I20260926 03:31:55.528173 6570 consensus_queue.cc:260] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } }
I20260926 03:31:55.528451 6570 raft_consensus.cc:388] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:31:55.528622 6570 raft_consensus.cc:482] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:31:55.528807 6570 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:55.530886 6570 raft_consensus.cc:504] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } }
I20260926 03:31:55.531374 6570 leader_election.cc:302] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aaf83112ebc946f5b71d630f6d3454b0; no voters:
I20260926 03:31:55.531971 6570 leader_election.cc:288] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:31:55.532135 6575 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:31:55.533257 6575 raft_consensus.cc:686] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [term 1 LEADER]: Becoming Leader. State: Replica: aaf83112ebc946f5b71d630f6d3454b0, State: Running, Role: LEADER
I20260926 03:31:55.533810 6575 consensus_queue.cc:237] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } }
I20260926 03:31:55.534133 6570 sys_catalog.cc:564] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:31:55.536932 6577 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aaf83112ebc946f5b71d630f6d3454b0. Latest consensus state: current_term: 1 leader_uuid: "aaf83112ebc946f5b71d630f6d3454b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } }
I20260926 03:31:55.537266 6577 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:55.537724 6576 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aaf83112ebc946f5b71d630f6d3454b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } }
I20260926 03:31:55.538028 6576 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:55.538223 6584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:31:55.542627 6584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:31:55.551218 6584 catalog_manager.cc:1347] Generated new cluster ID: 517c6d59af29429eb0c555bedc2befa8
I20260926 03:31:55.551323 6584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:31:55.608089 6584 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:31:55.609452 6584 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:31:55.623018 6584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0: Generated new TSK 0
I20260926 03:31:55.623908 6584 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260926 03:31:55.684527 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46591
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:56.026726 6596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:56.027115 6596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:56.027298 6596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:56.035812 6596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:56.036018 6596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:56.036203 6596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:31:56.046809 6596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:56.048098 6596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:56.049458 6596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:56.063076 6604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:56.063490 6602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:56.063887 6601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:56.064343 6596 server_base.cc:1034] running on GCE node
I20260926 03:31:56.065111 6596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:56.066474 6596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:56.067708 6596 hybrid_clock.cc:648] HybridClock initialized: now 1790393516067648 us; error 54 us; skew 500 ppm
I20260926 03:31:56.068056 6596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:56 dist-test-slave-dt7p krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393516, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:56.071295 6596 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260926 03:31:56.072729 6596 webserver.cc:460] Webserver started at http://127.0.97.1:40049/ using document root <none> and password file <none>
I20260926 03:31:56.073359 6596 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:56.073452 6596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:56.073691 6596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:56.076462 6596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-0/data/instance:
uuid: "6b0027955ae14788b8afa531cbfd18e1"
format_stamp: "Formatted at 2026-09-26 03:31:56 on dist-test-slave-dt7p"
server_key: "5624f1180d45ec01fddbb19db1605f99"
server_key_iv: "e813559505b187586949b15e5989fa78"
server_key_version: "encryptionkey@0"
I20260926 03:31:56.077289 6596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6b0027955ae14788b8afa531cbfd18e1"
format_stamp: "Formatted at 2026-09-26 03:31:56 on dist-test-slave-dt7p"
server_key: "5624f1180d45ec01fddbb19db1605f99"
server_key_iv: "e813559505b187586949b15e5989fa78"
server_key_version: "encryptionkey@0"
I20260926 03:31:56.082072 6596 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 03:31:56.085361 6611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:56.086706 6596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:31:56.086880 6596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-0/wal
uuid: "6b0027955ae14788b8afa531cbfd18e1"
format_stamp: "Formatted at 2026-09-26 03:31:56 on dist-test-slave-dt7p"
server_key: "5624f1180d45ec01fddbb19db1605f99"
server_key_iv: "e813559505b187586949b15e5989fa78"
server_key_version: "encryptionkey@0"
I20260926 03:31:56.087102 6596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:56.106282 6596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:56.110733 6596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:56.111050 6596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:56.112087 6596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:31:56.113547 6596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:31:56.113622 6596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:56.113776 6596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:31:56.113867 6596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:56.148800 6596 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41735
I20260926 03:31:56.148847 6723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41735 every 8 connection(s)
I20260926 03:31:56.150096 6596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:31:56.157430 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 6596
I20260926 03:31:56.157680 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:31:56.158394 388 external_mini_cluster.cc:1428] Setting key 7c0edb32276fc62bd7f19bb79b4a75b3
Sep 26 03:31:56 dist-test-slave-dt7p krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393516, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:31:56.171429 6726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:56.152700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53739 (local address 127.0.97.62:46591)
0926 03:31:56.153131 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:56.153135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:56.154112 (+ 977us) server_negotiation.cc:407] Connection header received
0926 03:31:56.155885 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:56.155890 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:56.155980 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:56.156105 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:56.157939 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:56.158756 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:56.160305 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:56.160785 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:56.160921 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:56.164918 (+ 3997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:56.164923 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:56.164926 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:56.164957 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:56.168102 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:56.168952 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:56.168960 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:56.168967 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:56.169106 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:56.169506 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:56.169514 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:56.169520 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:56.169900 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:56.170059 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:56.170719 (+ 660us) server_negotiation.cc:299] Negotiation successful
0926 03:31:56.170962 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":222,"threads_started":1}
I20260926 03:31:56.172061 6727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:56.153135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46591 (local address 127.0.97.1:53739)
0926 03:31:56.153936 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:56.153971 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:56.155622 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:56.156329 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:56.156336 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:56.156906 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:56.157744 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:56.157769 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:56.158928 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:56.158933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:56.160148 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:56.160159 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:56.161005 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:56.161011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:56.161231 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:56.161954 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:56.161987 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:56.164673 (+ 2686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:56.168264 (+ 3591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:56.168272 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:56.168285 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:56.168822 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:56.169213 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:56.169216 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:56.169218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:56.169385 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:56.170021 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:56.170026 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:56.170337 (+ 311us) client_negotiation.cc:769] Sending connection context
0926 03:31:56.170561 (+ 224us) client_negotiation.cc:240] Negotiation successful
0926 03:31:56.170944 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":387,"threads_started":1}
I20260926 03:31:56.174434 6724 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46591
I20260926 03:31:56.174785 6724 heartbeater.cc:464] Registering TS with master...
I20260926 03:31:56.175740 6724 heartbeater.cc:511] Master 127.0.97.62:46591 requested a full tablet report, sending...
I20260926 03:31:56.179050 6535 ts_manager.cc:194] Registered new tserver with Master: 6b0027955ae14788b8afa531cbfd18e1 (127.0.97.1:41735)
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 03:31:56.180742 6535 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53739
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260926 03:31:56.212962 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46591
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:56.556381 6730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:56.556731 6730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:56.556912 6730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:56.564481 6730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:56.564661 6730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:56.564859 6730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:31:56.574677 6730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:56.575969 6730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:56.577390 6730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:56.590474 6735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:56.590440 6736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:56.590509 6738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:56.591233 6730 server_base.cc:1034] running on GCE node
I20260926 03:31:56.591874 6730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:56.592873 6730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:56.594112 6730 hybrid_clock.cc:648] HybridClock initialized: now 1790393516594065 us; error 69 us; skew 500 ppm
I20260926 03:31:56.594439 6730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:56 dist-test-slave-dt7p krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393516, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:56.597594 6730 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260926 03:31:56.599031 6730 webserver.cc:460] Webserver started at http://127.0.97.2:35415/ using document root <none> and password file <none>
I20260926 03:31:56.599691 6730 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:56.599839 6730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:56.600148 6730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:56.602994 6730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-1/data/instance:
uuid: "b61e6b996c96424aa56ab30a31376891"
format_stamp: "Formatted at 2026-09-26 03:31:56 on dist-test-slave-dt7p"
server_key: "92b1bfa2587c0f6d71199fab9a624db7"
server_key_iv: "e2b931e3e93ccfe666fafd923287e6a4"
server_key_version: "encryptionkey@0"
I20260926 03:31:56.603771 6730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b61e6b996c96424aa56ab30a31376891"
format_stamp: "Formatted at 2026-09-26 03:31:56 on dist-test-slave-dt7p"
server_key: "92b1bfa2587c0f6d71199fab9a624db7"
server_key_iv: "e2b931e3e93ccfe666fafd923287e6a4"
server_key_version: "encryptionkey@0"
I20260926 03:31:56.608812 6730 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 03:31:56.612483 6745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:56.613765 6730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:31:56.613963 6730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-1/wal
uuid: "b61e6b996c96424aa56ab30a31376891"
format_stamp: "Formatted at 2026-09-26 03:31:56 on dist-test-slave-dt7p"
server_key: "92b1bfa2587c0f6d71199fab9a624db7"
server_key_iv: "e2b931e3e93ccfe666fafd923287e6a4"
server_key_version: "encryptionkey@0"
I20260926 03:31:56.614181 6730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:56.626995 6730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:56.630918 6730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:56.631211 6730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:56.632158 6730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:31:56.633526 6730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:31:56.633632 6730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:56.633781 6730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:31:56.633860 6730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:56.668179 6730 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34011
I20260926 03:31:56.668278 6857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34011 every 8 connection(s)
I20260926 03:31:56.669493 6730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:31:56.674775 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 6730
I20260926 03:31:56.675014 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:31:56.675593 388 external_mini_cluster.cc:1428] Setting key b89b9588725625475b33b581b048679d
Sep 26 03:31:56 dist-test-slave-dt7p krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393516, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:31:56.694367 6860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:56.672148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58353 (local address 127.0.97.62:46591)
0926 03:31:56.672658 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:56.672663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:56.673828 (+ 1165us) server_negotiation.cc:407] Connection header received
0926 03:31:56.676583 (+ 2755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:56.676590 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:56.676690 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:56.676824 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:56.678617 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:56.679333 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:56.682100 (+ 2767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:56.682524 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:56.682636 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:56.687139 (+ 4503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:56.687149 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:56.687157 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:56.687203 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:56.690771 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:56.691693 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:56.691701 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:56.691707 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:56.691854 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:56.692304 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:56.692312 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:56.692318 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:56.692743 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:56.692889 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:56.693618 (+ 729us) server_negotiation.cc:299] Negotiation successful
0926 03:31:56.693859 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":253,"threads_started":1}
I20260926 03:31:56.694981 6861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:56.672732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46591 (local address 127.0.97.2:58353)
0926 03:31:56.673658 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:56.673711 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:56.676224 (+ 2513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:56.677072 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:56.677081 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:56.677617 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:56.678400 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:56.678426 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:56.679524 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:56.679533 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:56.681937 (+ 2404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:56.681955 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:56.682780 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:56.682791 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:56.683144 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:56.683984 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:56.684021 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:56.686880 (+ 2859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:56.690926 (+ 4046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:56.690934 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:56.690949 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:56.691545 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:56.691975 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:56.691980 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:56.691982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:56.692167 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:56.692873 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:56.692880 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:56.693220 (+ 340us) client_negotiation.cc:769] Sending connection context
0926 03:31:56.693442 (+ 222us) client_negotiation.cc:240] Negotiation successful
0926 03:31:56.693817 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":388,"threads_started":1}
I20260926 03:31:56.696592 6858 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46591
I20260926 03:31:56.696926 6858 heartbeater.cc:464] Registering TS with master...
I20260926 03:31:56.697845 6858 heartbeater.cc:511] Master 127.0.97.62:46591 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 03:31:56.699447 6535 ts_manager.cc:194] Registered new tserver with Master: b61e6b996c96424aa56ab30a31376891 (127.0.97.2:34011)
I20260926 03:31:56.700368 6535 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58353
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260926 03:31:56.729089 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46591
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:57.045117 6864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:57.045380 6864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:57.045552 6864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:57.052845 6864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:57.052944 6864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:57.053128 6864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:31:57.062623 6864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:57.063910 6864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:57.065289 6864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:57.078452 6870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:57.078526 6869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:57.078502 6872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:57.079344 6864 server_base.cc:1034] running on GCE node
I20260926 03:31:57.079866 6864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:57.080741 6864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:57.081938 6864 hybrid_clock.cc:648] HybridClock initialized: now 1790393517081894 us; error 38 us; skew 500 ppm
I20260926 03:31:57.082202 6864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:57 dist-test-slave-dt7p krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393517, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:57.085139 6864 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260926 03:31:57.086581 6864 webserver.cc:460] Webserver started at http://127.0.97.3:37229/ using document root <none> and password file <none>
I20260926 03:31:57.087280 6864 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:57.087430 6864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:57.087728 6864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:57.090536 6864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-2/data/instance:
uuid: "91c53a144b0242e491f9e8b8aaef3f02"
format_stamp: "Formatted at 2026-09-26 03:31:57 on dist-test-slave-dt7p"
server_key: "38190125c2b2bb94e02b6ec37d5d707d"
server_key_iv: "ec38cc02266287a140de0d0ac165e39a"
server_key_version: "encryptionkey@0"
I20260926 03:31:57.091296 6864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-2/wal/instance:
uuid: "91c53a144b0242e491f9e8b8aaef3f02"
format_stamp: "Formatted at 2026-09-26 03:31:57 on dist-test-slave-dt7p"
server_key: "38190125c2b2bb94e02b6ec37d5d707d"
server_key_iv: "ec38cc02266287a140de0d0ac165e39a"
server_key_version: "encryptionkey@0"
I20260926 03:31:57.096050 6864 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260926 03:31:57.099493 6879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:57.100850 6864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:31:57.101025 6864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-2/wal
uuid: "91c53a144b0242e491f9e8b8aaef3f02"
format_stamp: "Formatted at 2026-09-26 03:31:57 on dist-test-slave-dt7p"
server_key: "38190125c2b2bb94e02b6ec37d5d707d"
server_key_iv: "ec38cc02266287a140de0d0ac165e39a"
server_key_version: "encryptionkey@0"
I20260926 03:31:57.101235 6864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:57.126395 6864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:57.130347 6864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:57.130654 6864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:57.131608 6864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:31:57.133067 6864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:31:57.133181 6864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:57.133333 6864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:31:57.133409 6864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:57.167506 6864 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36319
I20260926 03:31:57.167608 6991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36319 every 8 connection(s)
I20260926 03:31:57.168807 6864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:31:57.178514 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 6864
I20260926 03:31:57.178738 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:31:57.179147 388 external_mini_cluster.cc:1428] Setting key 12332b0fe89891beca0144e957775a57
Sep 26 03:31:57 dist-test-slave-dt7p krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393517, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:31:57.183692 6724 heartbeater.cc:503] Master 127.0.97.62:46591 was elected leader, sending a full tablet report...
I20260926 03:31:57.188845 6860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:57.171413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42937 (local address 127.0.97.62:46591)
0926 03:31:57.171599 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:57.171603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:57.172745 (+ 1142us) server_negotiation.cc:407] Connection header received
0926 03:31:57.174469 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:57.174475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:57.174580 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:57.174699 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:57.176138 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:57.176946 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:57.178452 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:57.178967 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:57.179112 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:57.182861 (+ 3749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:57.182866 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:57.182870 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:57.182901 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:57.185864 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:57.186603 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:57.186608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:57.186612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:57.186727 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:57.187159 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:57.187166 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:57.187170 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:57.187434 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:57.187596 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:57.188275 (+ 679us) server_negotiation.cc:299] Negotiation successful
0926 03:31:57.188495 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 03:31:57.189749 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:57.171848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46591 (local address 127.0.97.3:42937)
0926 03:31:57.172523 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:57.172624 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:57.174204 (+ 1580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:57.174924 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:57.174931 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:57.175407 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:57.175928 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:57.175944 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:57.177100 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:57.177106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:57.178301 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:57.178312 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:57.179185 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:57.179192 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:57.179418 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:57.180243 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:57.180265 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:57.182669 (+ 2404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:57.186016 (+ 3347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:57.186025 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:57.186042 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:57.186464 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:57.186842 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:57.186848 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:57.186851 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:57.187036 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:57.187576 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:57.187584 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:57.187913 (+ 329us) client_negotiation.cc:769] Sending connection context
0926 03:31:57.188190 (+ 277us) client_negotiation.cc:240] Negotiation successful
0926 03:31:57.188715 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":277,"threads_started":1}
I20260926 03:31:57.191126 6992 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46591
I20260926 03:31:57.191466 6992 heartbeater.cc:464] Registering TS with master...
I20260926 03:31:57.192158 6992 heartbeater.cc:511] Master 127.0.97.62:46591 requested a full tablet report, sending...
I20260926 03:31:57.193703 6535 ts_manager.cc:194] Registered new tserver with Master: 91c53a144b0242e491f9e8b8aaef3f02 (127.0.97.3:36319)
I20260926 03:31:57.194743 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:31:57.194813 6535 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42937
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260926 03:31:57.244439 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:42461
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46591,127.0.97.61:42461
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:31:57.568547 6997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:57.568848 6997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:57.569029 6997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:57.576001 6997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:31:57.576099 6997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:57.576197 6997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:57.576301 6997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:31:57.576393 6997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:31:57.585945 6997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46591,127.0.97.61:42461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:42461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
W20260926 03:31:57.586357 6997 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:46591,127.0.97.61:42461'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 03:31:57.587406 6997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:57.588702 6997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:57.600023 7003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:57.600039 7002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:57.600195 7005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:57.601107 6997 server_base.cc:1034] running on GCE node
I20260926 03:31:57.601841 6997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:57.603144 6997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:57.604357 6997 hybrid_clock.cc:648] HybridClock initialized: now 1790393517604307 us; error 26 us; skew 500 ppm
I20260926 03:31:57.604732 6997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:57 dist-test-slave-dt7p krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393517, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:57.607928 6997 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260926 03:31:57.609338 6997 webserver.cc:460] Webserver started at http://127.0.97.61:46477/ using document root <none> and password file <none>
I20260926 03:31:57.609925 6997 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:57.610005 6997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:57.610247 6997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:57.613006 6997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-1/data/instance:
uuid: "8692ab0d52964a07803500c4080166c4"
format_stamp: "Formatted at 2026-09-26 03:31:57 on dist-test-slave-dt7p"
server_key: "0cf1386b559631d80e55842940c6f76d"
server_key_iv: "ee02da92d608fd466e1677c986a209cd"
server_key_version: "encryptionkey@0"
I20260926 03:31:57.613816 6997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-1/wal/instance:
uuid: "8692ab0d52964a07803500c4080166c4"
format_stamp: "Formatted at 2026-09-26 03:31:57 on dist-test-slave-dt7p"
server_key: "0cf1386b559631d80e55842940c6f76d"
server_key_iv: "ee02da92d608fd466e1677c986a209cd"
server_key_version: "encryptionkey@0"
I20260926 03:31:57.618625 6997 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 03:31:57.621851 7012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:57.623075 6997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:31:57.623251 6997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-1/wal
uuid: "8692ab0d52964a07803500c4080166c4"
format_stamp: "Formatted at 2026-09-26 03:31:57 on dist-test-slave-dt7p"
server_key: "0cf1386b559631d80e55842940c6f76d"
server_key_iv: "ee02da92d608fd466e1677c986a209cd"
server_key_version: "encryptionkey@0"
I20260926 03:31:57.623452 6997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:57.644454 6997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:57.648710 6997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:57.648967 6997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:57.667002 6997 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:42461
I20260926 03:31:57.667049 7063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:42461 every 8 connection(s)
I20260926 03:31:57.668622 6997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-1/data/info.pb
I20260926 03:31:57.672132 7064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:57.673911 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 6997
I20260926 03:31:57.681259 7064 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:31:57 dist-test-slave-dt7p krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 26 03:31:57 dist-test-slave-dt7p krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393517, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:31:57.696944 7067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:57.676002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44152 (local address 127.0.97.61:42461)
0926 03:31:57.676794 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:57.676804 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:57.676840 (+ 36us) server_negotiation.cc:407] Connection header received
0926 03:31:57.678337 (+ 1497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:57.678356 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:57.678709 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:57.679121 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:57.680807 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:57.682126 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:57.683719 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:57.684820 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:57.685000 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:57.689564 (+ 4564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:57.689571 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:57.689581 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:57.689619 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:57.693138 (+ 3519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:57.693714 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:57.693722 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:57.693730 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:57.693864 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:57.694235 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:57.694242 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:57.694245 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:57.694695 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:57.694968 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:57.695412 (+ 444us) server_negotiation.cc:299] Negotiation successful
0926 03:31:57.695782 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":369,"threads_started":1}
I20260926 03:31:57.698396 6860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:57.683759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54302 (local address 127.0.97.62:46591)
0926 03:31:57.683981 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:57.683987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:57.684489 (+ 502us) server_negotiation.cc:407] Connection header received
0926 03:31:57.684669 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:57.684672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:57.684731 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:57.684806 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:57.685939 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:57.687106 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:57.688416 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:57.688933 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:57.689028 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:57.692885 (+ 3857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:57.692892 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:57.692900 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:57.692941 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:57.695549 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:57.696290 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:57.696295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:57.696298 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:57.696431 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:57.696862 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:57.696867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:57.696872 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:57.697299 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:57.697417 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:57.697950 (+ 533us) server_negotiation.cc:299] Negotiation successful
0926 03:31:57.698104 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 03:31:57.698633 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:57.683678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46591 (local address 127.0.0.1:54302)
0926 03:31:57.684334 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:57.684384 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:57.684523 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:57.684934 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:57.684941 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:57.685565 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:57.685790 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:57.685798 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:57.687295 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:57.687301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:57.688272 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:57.688282 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:57.689154 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:57.689163 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:57.689353 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:57.690085 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:57.690101 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:57.692703 (+ 2602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:57.695757 (+ 3054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:57.695766 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:57.695774 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:57.696168 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:57.696541 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:57.696544 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:57.696546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:57.696758 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:57.697430 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:57.697437 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:57.697643 (+ 206us) client_negotiation.cc:769] Sending connection context
0926 03:31:57.697832 (+ 189us) client_negotiation.cc:240] Negotiation successful
0926 03:31:57.698112 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":296,"threads_started":1}
I20260926 03:31:57.700336 7064 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42461 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:31:57.702975 6858 heartbeater.cc:503] Master 127.0.97.62:46591 was elected leader, sending a full tablet report...
Sep 26 03:31:57 dist-test-slave-dt7p krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393517, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260926 03:31:57.712333 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:57.701165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42461 (local address 127.0.0.1:44156)
0926 03:31:57.701319 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:57.701334 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:57.701404 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:57.701700 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:57.701703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:57.701974 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:57.702088 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:57.702099 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:57.702994 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:57.703002 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:57.704090 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:57.704103 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:57.705039 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:57.705043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:57.705199 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:57.705739 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:57.705754 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:57.707711 (+ 1957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:57.710411 (+ 2700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:57.710415 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:57.710418 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:57.710762 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:57.711062 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:57.711065 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:57.711067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:57.711180 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:57.711655 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:57.711658 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:57.711782 (+ 124us) client_negotiation.cc:769] Sending connection context
0926 03:31:57.711890 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 03:31:57.712086 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 03:31:57.712436 7067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:57.701327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44156 (local address 127.0.97.61:42461)
0926 03:31:57.701494 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:57.701498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:57.701511 (+ 13us) server_negotiation.cc:407] Connection header received
0926 03:31:57.701545 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:57.701548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:57.701595 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:57.701668 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:57.702214 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:57.702867 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:57.704249 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:57.704915 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:57.705018 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:57.707884 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:57.707889 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:57.707892 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:57.707918 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:57.710293 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:57.710861 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:57.710865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:57.710867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:57.710968 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:57.711268 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:57.711271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:57.711273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:57.711532 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:57.711630 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:57.711977 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 03:31:57.712204 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":59}
I20260926 03:31:57.716987 7064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4: Bootstrap starting.
I20260926 03:31:57.719887 7064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:57.720944 7064 log.cc:824] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4: Log is configured to *not* fsync() on all Append() calls
I20260926 03:31:57.723359 7064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4: No bootstrap required, opened a new log
I20260926 03:31:57.730226 7064 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42461 } }
I20260926 03:31:57.730571 7064 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:57.730664 7064 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8692ab0d52964a07803500c4080166c4, State: Initialized, Role: FOLLOWER
I20260926 03:31:57.731228 7064 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42461 } }
I20260926 03:31:57.732007 7070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42461 } } }
I20260926 03:31:57.732336 7070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:31:57.733001 7064 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 03:31:57.741415 7081 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:31:57.741566 7081 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:31:57.744117 6997 master_runner.cc:428] Detected that this master 8692ab0d52964a07803500c4080166c4 is joining an existing cluster
I20260926 03:31:57.744257 6997 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:42461
I20260926 03:31:57.766729 7067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:57.748248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44170 (local address 127.0.97.61:42461)
0926 03:31:57.748471 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:57.748478 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:57.749549 (+ 1071us) server_negotiation.cc:407] Connection header received
0926 03:31:57.749661 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:57.749666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:57.749727 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:57.749816 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:57.751414 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:57.752548 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:57.756846 (+ 4298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:57.757629 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:57.757748 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:57.760123 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:57.760129 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:57.760133 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:57.760171 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:57.763574 (+ 3403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:57.764527 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:57.764532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:57.764536 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:57.764694 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:57.765128 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:57.765133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:57.765136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:57.765478 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:57.765616 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:57.766120 (+ 504us) server_negotiation.cc:299] Negotiation successful
0926 03:31:57.766340 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64,"spinlock_wait_cycles":27008}
I20260926 03:31:57.767349 6860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:57.748630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54316 (local address 127.0.97.62:46591)
0926 03:31:57.748812 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:57.748818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:57.749199 (+ 381us) server_negotiation.cc:407] Connection header received
0926 03:31:57.749484 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:57.749488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:57.749577 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:57.749664 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:57.751415 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:57.752547 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:57.754673 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:57.755289 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:57.755404 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:57.759571 (+ 4167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:57.759580 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:57.759587 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:57.759631 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:57.763606 (+ 3975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:57.764532 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:57.764540 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:57.764546 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:57.764717 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:57.765204 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:57.765212 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:57.765218 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:57.765589 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:57.765707 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:57.766338 (+ 631us) server_negotiation.cc:299] Negotiation successful
0926 03:31:57.766570 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 03:31:57.767730 7087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:57.748257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42461 (local address 127.0.0.1:44170)
0926 03:31:57.749425 (+ 1168us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:57.749442 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:57.749557 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:57.749853 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:57.749858 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:57.750466 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:57.751284 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:57.751297 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:57.752769 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:57.752774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:57.756675 (+ 3901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:57.756691 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:57.758019 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:57.758025 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:57.758203 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:57.758817 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:57.758839 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:57.759948 (+ 1109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:57.763711 (+ 3763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:57.763719 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:57.763724 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:57.764396 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:57.764819 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:57.764825 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:57.764828 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:57.765008 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:57.765694 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:57.765698 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:57.765842 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 03:31:57.766989 (+ 1147us) client_negotiation.cc:240] Negotiation successful
0926 03:31:57.767272 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1011,"mutex_wait_us":157,"spinlock_wait_cycles":18304,"thread_start_us":455,"threads_started":1}
I20260926 03:31:57.769039 7088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:57.748096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46591 (local address 127.0.0.1:54316)
0926 03:31:57.749085 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:57.749114 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:57.749306 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:57.749713 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:57.749718 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:57.750463 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:57.751219 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:57.751241 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:57.752769 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:57.752774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:57.754503 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:57.754519 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:57.755683 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:57.755690 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:57.755854 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:57.757718 (+ 1864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:57.757829 (+ 111us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:57.759353 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:57.763767 (+ 4414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:57.763779 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:57.763785 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:57.764391 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:57.764860 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:57.764870 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:57.764877 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:57.765078 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:57.765800 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:57.765808 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:57.766048 (+ 240us) client_negotiation.cc:769] Sending connection context
0926 03:31:57.768076 (+ 2028us) client_negotiation.cc:240] Negotiation successful
0926 03:31:57.768433 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"mutex_wait_us":132,"spinlock_wait_cycles":12416,"thread_start_us":1001,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Sep 26 03:31:57 dist-test-slave-dt7p krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393515, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260926 03:31:57.795655 7067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:57.775670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44174 (local address 127.0.97.61:42461)
0926 03:31:57.775825 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:57.775831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:57.778513 (+ 2682us) server_negotiation.cc:407] Connection header received
0926 03:31:57.778822 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:57.778827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:57.778940 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:57.779046 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:57.780214 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:57.781330 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:57.783198 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:57.784004 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:57.784140 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:57.788119 (+ 3979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:57.788128 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:57.788136 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:57.788181 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:57.792217 (+ 4036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:57.793080 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:57.793088 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:57.793093 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:57.793267 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:57.793708 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:57.793715 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:57.793722 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:57.794136 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:57.794314 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:57.795060 (+ 746us) server_negotiation.cc:299] Negotiation successful
0926 03:31:57.795294 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 03:31:57.795985 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:57.775635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42461 (local address 127.0.0.1:44174)
0926 03:31:57.778352 (+ 2717us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:57.778399 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:57.778605 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:57.779102 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:57.779108 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:57.779779 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:57.780067 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:57.780078 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:57.781487 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:57.781493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:57.783028 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:57.783044 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:57.784199 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:57.784206 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:57.784415 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:57.785148 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:57.785171 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:57.787921 (+ 2750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:57.792361 (+ 4440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:57.792372 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:57.792380 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:57.792947 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:57.793397 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:57.793402 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:57.793406 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:57.793568 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:57.794399 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:57.794420 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:57.794631 (+ 211us) client_negotiation.cc:769] Sending connection context
0926 03:31:57.794857 (+ 226us) client_negotiation.cc:240] Negotiation successful
0926 03:31:57.795233 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2496,"thread_start_us":2331,"threads_started":1}
I20260926 03:31:57.798329 6535 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:42461: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8692ab0d52964a07803500c4080166c4"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 42461
}
attrs {
promote: true
}
}
dest_uuid: "aaf83112ebc946f5b71d630f6d3454b0"
cas_config_opid_index: -1
I20260926 03:31:57.799480 6535 consensus_queue.cc:237] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: true } }
I20260926 03:31:57.802743 7094 raft_consensus.cc:2932] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 8692ab0d52964a07803500c4080166c4 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: true } } }
I20260926 03:31:57.803853 7039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:57.807494 7039 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [term 1 FOLLOWER]: Refusing update from remote peer aaf83112ebc946f5b71d630f6d3454b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 03:31:57.807507 7094 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:42461
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260926 03:31:57.807879 7094 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "aaf83112ebc946f5b71d630f6d3454b0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: true } } }
I20260926 03:31:57.808074 7094 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:57.808228 6997 master.cc:537] Master@127.0.97.61:42461 shutting down...
I20260926 03:31:57.809290 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:36639
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46591,127.0.97.61:42461,127.0.97.60:36639
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:31:57.826591 6997 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:31:57.827201 6997 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:31:57.827486 6997 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4: stopping tablet replica
I20260926 03:31:57.825510 7093 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.006s
I20260926 03:31:57.836966 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:57.836383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42461 (local address 127.0.0.1:44180)
0926 03:31:57.836652 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:57.836760 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260926 03:31:57.837317 6519 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42461: connect: Connection refused (error 111)
I20260926 03:31:57.839299 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:57.838823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42461 (local address 127.0.0.1:44190)
0926 03:31:57.839039 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:57.839132 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260926 03:31:57.839507 6519 consensus_peers.cc:487] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 -> Peer 8692ab0d52964a07803500c4080166c4 (127.0.97.61:42461): Couldn't send request to peer 8692ab0d52964a07803500c4080166c4. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42461: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 03:31:57.839785 6997 master.cc:552] Master@127.0.97.61:42461 shutdown complete.
I20260926 03:31:57.839855 6997 master_runner.cc:305] Clearing existing system tablet
I20260926 03:31:57.840251 6997 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:57.840358 6997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:57.843362 6997 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260926 03:31:57.845156 7099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:57.845570 6997 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:57.845862 6997 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 03:31:57.845944 6997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-1/wal
uuid: "8692ab0d52964a07803500c4080166c4"
format_stamp: "Formatted at 2026-09-26 03:31:57 on dist-test-slave-dt7p"
server_key: "0cf1386b559631d80e55842940c6f76d"
server_key_iv: "ee02da92d608fd466e1677c986a209cd"
server_key_version: "encryptionkey@0"
I20260926 03:31:57.850828 6997 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:31:57.857710 6997 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 03:31:57.857882 6997 log.cc:1196] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 03:31:57.858322 6997 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4: Deleting consensus metadata
I20260926 03:31:57.858597 6997 master_runner.cc:315] Copying system tablet from 127.0.97.62:46591
I20260926 03:31:57.866026 6997 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:46591
I20260926 03:31:57.882025 6860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:57.866948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54322 (local address 127.0.97.62:46591)
0926 03:31:57.867145 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:57.867154 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:57.867703 (+ 549us) server_negotiation.cc:407] Connection header received
0926 03:31:57.867896 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:57.867901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:57.867984 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:57.868101 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:57.869203 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:57.870365 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:57.872357 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:57.873024 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:57.873159 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:57.875279 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:57.875286 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:57.875291 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:57.875330 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:57.878865 (+ 3535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:57.879842 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:57.879848 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:57.879852 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:57.879984 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:57.880407 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:57.880412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:57.880415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:57.880790 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:57.880939 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:57.881459 (+ 520us) server_negotiation.cc:299] Negotiation successful
0926 03:31:57.881672 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:31:57.882061 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:57.866746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46591 (local address 127.0.0.1:54322)
0926 03:31:57.867561 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:57.867585 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:57.867759 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:57.868150 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:57.868154 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:57.868746 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:57.869039 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:57.869052 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:57.870531 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:57.870536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:57.872188 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:57.872203 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:57.873189 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:57.873195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:57.873376 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:57.874042 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:57.874066 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:57.875075 (+ 1009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:57.879024 (+ 3949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:57.879032 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:57.879039 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:57.879718 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:57.880115 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:57.880120 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:57.880123 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:57.880289 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:57.880916 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:57.880973 (+ 57us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:57.881149 (+ 176us) client_negotiation.cc:769] Sending connection context
0926 03:31:57.881307 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 03:31:57.881617 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":539,"threads_started":1}
I20260926 03:31:57.883391 6555 tablet_copy_service.cc:140] P aaf83112ebc946f5b71d630f6d3454b0: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8692ab0d52964a07803500c4080166c4 ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:54322)
I20260926 03:31:57.883657 6555 tablet_copy_service.cc:161] P aaf83112ebc946f5b71d630f6d3454b0: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8692ab0d52964a07803500c4080166c4 at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:54322: session id = 8692ab0d52964a07803500c4080166c4-00000000000000000000000000000000
I20260926 03:31:57.888123 6555 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 03:31:57.890242 6997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 03:31:57.896901 6997 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4: tablet copy: Starting download of 0 data blocks...
I20260926 03:31:57.897229 6997 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4: tablet copy: Starting download of 1 WAL segments...
I20260926 03:31:57.899860 6997 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 03:31:57.904459 6555 tablet_copy_service.cc:342] P aaf83112ebc946f5b71d630f6d3454b0: Request end of tablet copy session 8692ab0d52964a07803500c4080166c4-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:54322
I20260926 03:31:57.904671 6555 tablet_copy_service.cc:433] P aaf83112ebc946f5b71d630f6d3454b0: ending tablet copy session 8692ab0d52964a07803500c4080166c4-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8692ab0d52964a07803500c4080166c4
I20260926 03:31:57.916323 6997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:57.919245 7106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:57.925453 7107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:57.932257 7109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:57.932699 6997 server_base.cc:1034] running on GCE node
I20260926 03:31:57.933058 6997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:57.933756 6997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:57.935083 6997 hybrid_clock.cc:648] HybridClock initialized: now 1790393517935026 us; error 33 us; skew 500 ppm
I20260926 03:31:57.935379 6997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:57 dist-test-slave-dt7p krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393517, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:57.937997 6997 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260926 03:31:57.939136 6997 webserver.cc:460] Webserver started at http://127.0.97.61:33313/ using document root <none> and password file <none>
I20260926 03:31:57.939478 6997 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:57.939558 6997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:57.942610 6997 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260926 03:31:57.944589 7116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:57.945082 6997 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260926 03:31:57.945230 6997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-1/wal
uuid: "8692ab0d52964a07803500c4080166c4"
format_stamp: "Formatted at 2026-09-26 03:31:57 on dist-test-slave-dt7p"
server_key: "0cf1386b559631d80e55842940c6f76d"
server_key_iv: "ee02da92d608fd466e1677c986a209cd"
server_key_version: "encryptionkey@0"
I20260926 03:31:57.945387 6997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:57.970549 6997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:57.974632 6997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:57.995855 6997 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:42461
I20260926 03:31:57.995879 7167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:42461 every 8 connection(s)
I20260926 03:31:57.996832 6997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-1/data/info.pb
I20260926 03:31:58.001929 7168 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:31:58.005861 7168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4: Bootstrap starting.
I20260926 03:31:58.021358 7168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:31:58.021834 7168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4: Bootstrap complete.
I20260926 03:31:58.022771 7168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: true } }
I20260926 03:31:58.023013 7168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8692ab0d52964a07803500c4080166c4, State: Initialized, Role: LEARNER
I20260926 03:31:58.023247 7168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: true } }
I20260926 03:31:58.024242 7168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:31:58.024044 7171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: true } } }
I20260926 03:31:58.024489 7171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [sys.catalog]: This master's current role is: LEARNER
I20260926 03:31:58.031973 7182 catalog_manager.cc:1259] Loaded cluster ID: 517c6d59af29429eb0c555bedc2befa8
I20260926 03:31:58.032096 7182 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4: loading cluster ID for follower catalog manager: success
I20260926 03:31:58.035619 7182 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4: acquiring CA information for follower catalog manager: success
I20260926 03:31:58.038501 7182 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 03:31:58.197741 6992 heartbeater.cc:503] Master 127.0.97.62:46591 was elected leader, sending a full tablet report...
W20260926 03:31:58.207419 7096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:58.207703 7096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:58.207867 7096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:58.215523 7096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:31:58.215646 7096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:58.215726 7096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:58.215807 7096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:31:58.215863 7096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:31:58.227509 7096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46591,127.0.97.61:42461,127.0.97.60:36639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:36639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:58.228855 7096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:58.229202 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:58.222587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42461 (local address 127.0.0.1:44192)
0926 03:31:58.222762 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:58.222787 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:58.222885 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:58.223649 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:58.223656 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:58.223683 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:31:58.223893 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:58.223902 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.225354 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.225361 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:58.227035 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:58.227048 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.228283 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.228290 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:58.228505 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:58.228600 (+ 95us) client_negotiation.cc:769] Sending connection context
0926 03:31:58.228752 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 03:31:58.228898 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 03:31:58.229219 7186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:58.222771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44192 (local address 127.0.97.61:42461)
0926 03:31:58.223321 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:58.223326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:58.223343 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:31:58.223399 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:58.223403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:58.223478 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:58.223613 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:31:58.224061 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.225214 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.227193 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.228115 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.228254 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:58.228494 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:58.228773 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 03:31:58.228901 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":324,"threads_started":1}
I20260926 03:31:58.230545 7096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 03:31:58.230525 7143 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260926 03:31:58.231086 7171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aaf83112ebc946f5b71d630f6d3454b0. Latest consensus state: current_term: 1 leader_uuid: "aaf83112ebc946f5b71d630f6d3454b0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: true } } }
I20260926 03:31:58.231369 7171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [sys.catalog]: This master's current role is: LEARNER
W20260926 03:31:58.242749 7188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:58.242713 7189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:58.242792 7191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:58.246084 7096 server_base.cc:1034] running on GCE node
I20260926 03:31:58.246847 7096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:58.248143 7096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:58.249369 7096 hybrid_clock.cc:648] HybridClock initialized: now 1790393518249334 us; error 28 us; skew 500 ppm
I20260926 03:31:58.249740 7096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:58 dist-test-slave-dt7p krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393518, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:58.253160 7096 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260926 03:31:58.254557 7096 webserver.cc:460] Webserver started at http://127.0.97.60:45187/ using document root <none> and password file <none>
I20260926 03:31:58.255216 7096 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:58.255364 7096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:58.255661 7096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:31:58.258540 7096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-2/data/instance:
uuid: "5a56fb218c4e482fac40da5c0be63a91"
format_stamp: "Formatted at 2026-09-26 03:31:58 on dist-test-slave-dt7p"
server_key: "55e7addc37616f6f7a794a31a3c34f9b"
server_key_iv: "7973e310ec5b297806b806e9a02c8f26"
server_key_version: "encryptionkey@0"
I20260926 03:31:58.259346 7096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-2/wal/instance:
uuid: "5a56fb218c4e482fac40da5c0be63a91"
format_stamp: "Formatted at 2026-09-26 03:31:58 on dist-test-slave-dt7p"
server_key: "55e7addc37616f6f7a794a31a3c34f9b"
server_key_iv: "7973e310ec5b297806b806e9a02c8f26"
server_key_version: "encryptionkey@0"
I20260926 03:31:58.264042 7096 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 03:31:58.267267 7198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:58.268432 7096 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 03:31:58.268656 7096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-2/wal
uuid: "5a56fb218c4e482fac40da5c0be63a91"
format_stamp: "Formatted at 2026-09-26 03:31:58 on dist-test-slave-dt7p"
server_key: "55e7addc37616f6f7a794a31a3c34f9b"
server_key_iv: "7973e310ec5b297806b806e9a02c8f26"
server_key_version: "encryptionkey@0"
I20260926 03:31:58.268846 7096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:58.292395 7096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:58.296130 7096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:58.296417 7096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:58.316212 7096 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:36639
I20260926 03:31:58.316241 7249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:36639 every 8 connection(s)
I20260926 03:31:58.317725 7096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-2/data/info.pb
I20260926 03:31:58.321442 7250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:31:58.321762 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 7096
I20260926 03:31:58.331641 7250 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:31:58 dist-test-slave-dt7p krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 26 03:31:58 dist-test-slave-dt7p krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393518, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:31:58.346151 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:58.323473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36108 (local address 127.0.97.60:36639)
0926 03:31:58.324219 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:58.324229 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:58.324266 (+ 37us) server_negotiation.cc:407] Connection header received
0926 03:31:58.326391 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:58.326410 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:58.326801 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:58.327096 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:58.328892 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.330423 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.332144 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.332902 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.333019 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:58.338079 (+ 5060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:58.338085 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:58.338094 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:58.338129 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:58.341593 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:58.342626 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:58.342634 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:58.342642 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:58.342800 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:58.343247 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:58.343253 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:58.343257 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:58.343730 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:58.344011 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:58.344476 (+ 465us) server_negotiation.cc:299] Negotiation successful
0926 03:31:58.344911 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":350,"threads_started":1}
I20260926 03:31:58.349865 6860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:58.334762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54338 (local address 127.0.97.62:46591)
0926 03:31:58.335054 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:58.335063 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:58.335629 (+ 566us) server_negotiation.cc:407] Connection header received
0926 03:31:58.335804 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:58.335808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:58.335877 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:58.335954 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:58.337291 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.338267 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.339929 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.340315 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.340465 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:58.343806 (+ 3341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:58.343810 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:58.343814 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:58.343844 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:58.347110 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:58.347821 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:58.347829 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:58.347835 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:58.347959 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:58.348370 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:58.348377 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:58.348380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:58.348745 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:58.348914 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:58.349346 (+ 432us) server_negotiation.cc:299] Negotiation successful
0926 03:31:58.349550 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 03:31:58.349937 7254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:58.334652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46591 (local address 127.0.0.1:54338)
0926 03:31:58.335472 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:58.335525 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:58.335685 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:58.336051 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:58.336060 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:58.336832 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:58.337116 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:58.337127 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.338411 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.338417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:58.339772 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:58.339783 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.340509 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.340515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:58.340721 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:58.341314 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:58.341331 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:58.343604 (+ 2273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:58.347248 (+ 3644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:58.347260 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:58.347267 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:58.347691 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:58.348076 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:58.348082 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:58.348086 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:58.348211 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:58.348876 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:58.348884 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:58.349056 (+ 172us) client_negotiation.cc:769] Sending connection context
0926 03:31:58.349247 (+ 191us) client_negotiation.cc:240] Negotiation successful
0926 03:31:58.349550 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":429,"threads_started":1}
I20260926 03:31:58.351064 7250 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42461 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:31:58 dist-test-slave-dt7p krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393518, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260926 03:31:58.363271 7254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:58.351677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42461 (local address 127.0.0.1:44194)
0926 03:31:58.351859 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:58.351878 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:58.351992 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:58.352381 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:58.352385 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:58.352769 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:58.352887 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:58.352894 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.353843 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.353851 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:58.354741 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:58.354758 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.355468 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.355473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:58.355662 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:58.356229 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:58.356245 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:58.358180 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:58.360991 (+ 2811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:58.360997 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:58.361001 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:58.361343 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:58.361698 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:58.361705 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:58.361708 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:58.361835 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:58.362333 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:58.362337 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:58.362478 (+ 141us) client_negotiation.cc:769] Sending connection context
0926 03:31:58.362634 (+ 156us) client_negotiation.cc:240] Negotiation successful
0926 03:31:58.362954 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 03:31:58.363286 7186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:58.351876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44194 (local address 127.0.97.61:42461)
0926 03:31:58.352066 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:58.352073 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:58.352098 (+ 25us) server_negotiation.cc:407] Connection header received
0926 03:31:58.352154 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:58.352157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:58.352214 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:58.352339 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:58.353024 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.353714 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.354893 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.355317 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.355459 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:58.358364 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:58.358369 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:58.358372 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:58.358401 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:58.360862 (+ 2461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:58.361461 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:58.361468 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:58.361471 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:58.361584 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:58.361948 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:58.361953 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:58.361956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:58.362214 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:58.362365 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:58.362723 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 03:31:58.362936 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 03:31:58.364327 7250 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36639 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:31:58 dist-test-slave-dt7p krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393518, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260926 03:31:58.376297 7254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:58.365015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36639 (local address 127.0.0.1:36120)
0926 03:31:58.365174 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:58.365189 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:58.365258 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:58.365564 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:58.365567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:58.365851 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:58.365979 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:58.365986 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.366870 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.366874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:58.368054 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:58.368064 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.368897 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.368902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:58.369097 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:58.369605 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:58.369621 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:58.371494 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:58.374323 (+ 2829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:58.374327 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:58.374330 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:58.374687 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:58.374976 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:58.374980 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:58.374982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:58.375115 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:58.375591 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:58.375595 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:58.375708 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 03:31:58.375826 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 03:31:58.376019 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":21}
I20260926 03:31:58.376422 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:58.365167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36120 (local address 127.0.97.60:36639)
0926 03:31:58.365328 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:58.365332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:58.365346 (+ 14us) server_negotiation.cc:407] Connection header received
0926 03:31:58.365403 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:58.365406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:58.365454 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:58.365530 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:58.366108 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.366759 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.368225 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.368736 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.368871 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:58.371649 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:58.371654 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:58.371657 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:58.371682 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:58.374214 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:58.374785 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:58.374789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:58.374791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:58.374882 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:58.375209 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:58.375212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:58.375214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:58.375464 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:58.375566 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:58.375905 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 03:31:58.376147 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":98,"server-negotiator.queue_time_us":75}
I20260926 03:31:58.380918 7250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91: Bootstrap starting.
I20260926 03:31:58.383836 7250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91: Neither blocks nor log segments found. Creating new log.
I20260926 03:31:58.384809 7250 log.cc:824] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91: Log is configured to *not* fsync() on all Append() calls
I20260926 03:31:58.388396 7250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91: No bootstrap required, opened a new log
I20260926 03:31:58.395421 7250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42461 } } peers { permanent_uuid: "5a56fb218c4e482fac40da5c0be63a91" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36639 } }
I20260926 03:31:58.395812 7250 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:31:58.395903 7250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a56fb218c4e482fac40da5c0be63a91, State: Initialized, Role: FOLLOWER
I20260926 03:31:58.396369 7250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42461 } } peers { permanent_uuid: "5a56fb218c4e482fac40da5c0be63a91" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36639 } }
I20260926 03:31:58.397177 7256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42461 } } peers { permanent_uuid: "5a56fb218c4e482fac40da5c0be63a91" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36639 } } }
I20260926 03:31:58.397462 7256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:31:58.398147 7250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 03:31:58.407516 7267 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:31:58.407651 7267 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:31:58.411552 7096 master_runner.cc:428] Detected that this master 5a56fb218c4e482fac40da5c0be63a91 is joining an existing cluster
I20260926 03:31:58.411643 7096 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:36639
I20260926 03:31:58.435448 7186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:58.416608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44206 (local address 127.0.97.61:42461)
0926 03:31:58.416853 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:58.416862 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:58.417298 (+ 436us) server_negotiation.cc:407] Connection header received
0926 03:31:58.417543 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:58.417550 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:58.417629 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:58.417697 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:58.419501 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.420993 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.424125 (+ 3132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.425042 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.425181 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:58.428365 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:58.428372 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:58.428377 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:58.428415 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:58.432265 (+ 3850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:58.433022 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:58.433027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:58.433030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:58.433170 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:58.433655 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:58.433660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:58.433663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:58.433994 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:58.434117 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:58.434850 (+ 733us) server_negotiation.cc:299] Negotiation successful
0926 03:31:58.435079 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260926 03:31:58.435767 7273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:58.416041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42461 (local address 127.0.0.1:44206)
0926 03:31:58.417216 (+ 1175us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:58.417237 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:58.417392 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:58.417962 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:58.417969 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:58.418839 (+ 870us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:58.419306 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:58.419321 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.421213 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.421221 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:58.423884 (+ 2663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:58.423903 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.425571 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.425581 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:58.425831 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:58.426741 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:58.426767 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:58.428176 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:58.432399 (+ 4223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:58.432405 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:58.432410 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:58.432897 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:58.433314 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:58.433322 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:58.433327 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:58.433511 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:58.434204 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:58.434269 (+ 65us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:58.434550 (+ 281us) client_negotiation.cc:769] Sending connection context
0926 03:31:58.434731 (+ 181us) client_negotiation.cc:240] Negotiation successful
0926 03:31:58.435070 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1022,"spinlock_wait_cycles":20352,"thread_start_us":11019,"threads_started":1}
I20260926 03:31:58.444366 6860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:58.415855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54344 (local address 127.0.97.62:46591)
0926 03:31:58.416058 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:58.416065 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:58.427461 (+ 11396us) server_negotiation.cc:407] Connection header received
0926 03:31:58.427658 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:58.427664 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:58.427743 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:58.427847 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:58.429295 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.430397 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.433004 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.433674 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.433764 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:58.437344 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:58.437351 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:58.437355 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:58.437393 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:58.441233 (+ 3840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:58.442060 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:58.442068 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:58.442080 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:58.442232 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:58.442689 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:58.442694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:58.442698 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:58.443027 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:58.443155 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:58.443761 (+ 606us) server_negotiation.cc:299] Negotiation successful
0926 03:31:58.443997 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 03:31:58.444708 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:58.417659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36126 (local address 127.0.97.60:36639)
0926 03:31:58.417867 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:58.417876 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:58.427932 (+ 10056us) server_negotiation.cc:407] Connection header received
0926 03:31:58.428123 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:58.428127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:58.428185 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:58.428248 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:58.429689 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.430881 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.433460 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.434204 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.434349 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:58.438725 (+ 4376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:58.438731 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:58.438735 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:58.438768 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:58.441832 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:58.442574 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:58.442583 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:58.442588 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:58.442707 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:58.443067 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:58.443078 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:58.443082 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:58.443374 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:58.443547 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:58.444144 (+ 597us) server_negotiation.cc:299] Negotiation successful
0926 03:31:58.444339 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":40}
I20260926 03:31:58.444844 7275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:58.416063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46591 (local address 127.0.0.1:54344)
0926 03:31:58.427354 (+ 11291us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:58.427374 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:58.427503 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:58.428017 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:58.428024 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:58.428760 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:58.429115 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:58.429127 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.430595 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.430600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:58.432839 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:58.432853 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.434042 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.434049 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:58.434292 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:58.435892 (+ 1600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:58.435913 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:58.437140 (+ 1227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:58.441382 (+ 4242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:58.441387 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:58.441391 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:58.441949 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:58.442357 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:58.442362 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:58.442366 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:58.442552 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:58.443229 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:58.443234 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:58.443448 (+ 214us) client_negotiation.cc:769] Sending connection context
0926 03:31:58.443636 (+ 188us) client_negotiation.cc:240] Negotiation successful
0926 03:31:58.443941 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11148,"mutex_wait_us":711,"thread_start_us":851,"threads_started":1}
I20260926 03:31:58.445722 7274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:58.417339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36639 (local address 127.0.0.1:36126)
0926 03:31:58.427858 (+ 10519us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:58.427874 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:58.428006 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:58.428438 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:58.428442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:58.429158 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:58.429540 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:58.429553 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.431112 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.431117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:58.433280 (+ 2163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:58.433298 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.434450 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.434457 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:58.434611 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:58.437543 (+ 2932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:58.437562 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:58.438532 (+ 970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:58.441971 (+ 3439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:58.441975 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:58.441979 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:58.442451 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:58.442825 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:58.442830 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:58.442834 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:58.442947 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:58.443495 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:58.443500 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:58.443669 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 03:31:58.444968 (+ 1299us) client_negotiation.cc:240] Negotiation successful
0926 03:31:58.445284 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10377,"thread_start_us":9297,"threads_started":1}
I20260926 03:31:58.451707 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 6596
Sep 26 03:31:58 dist-test-slave-dt7p krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393515, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 26 03:31:58 dist-test-slave-dt7p krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260926 03:31:58.465724 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:58.450107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36134 (local address 127.0.97.60:36639)
0926 03:31:58.450259 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:58.450264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:58.450277 (+ 13us) server_negotiation.cc:407] Connection header received
0926 03:31:58.450343 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:58.450346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:58.450431 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:58.450558 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:58.451302 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.452200 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.453887 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.454499 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.454646 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:58.458768 (+ 4122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:58.458775 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:58.458780 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:58.458817 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:58.462978 (+ 4161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:58.463802 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:58.463810 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:58.463814 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:58.463947 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:58.464319 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:58.464327 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:58.464330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:58.464666 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:58.464793 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:58.465192 (+ 399us) server_negotiation.cc:299] Negotiation successful
0926 03:31:58.465396 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 03:31:58.465739 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:58.449831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36639 (local address 127.0.0.1:36134)
0926 03:31:58.450048 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:58.450067 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:58.450165 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:58.450580 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:58.450585 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:58.451022 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:58.451188 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:58.451199 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.452344 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.452350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:58.453739 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:58.453754 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.454667 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.454673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:58.455299 (+ 626us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:58.455904 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:58.455957 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:58.458562 (+ 2605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:58.463124 (+ 4562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:58.463131 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:58.463135 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:58.463668 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:58.464077 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:58.464083 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:58.464087 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:58.464209 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:58.464776 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:58.464780 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:58.464946 (+ 166us) client_negotiation.cc:769] Sending connection context
0926 03:31:58.465099 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 03:31:58.465359 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 03:31:58.467509 6535 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:36639: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "5a56fb218c4e482fac40da5c0be63a91"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 36639
}
attrs {
promote: true
}
}
dest_uuid: "aaf83112ebc946f5b71d630f6d3454b0"
cas_config_opid_index: 5
I20260926 03:31:58.468446 6535 consensus_queue.cc:237] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: true } } peers { permanent_uuid: "5a56fb218c4e482fac40da5c0be63a91" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36639 } attrs { promote: true } }
I20260926 03:31:58.470089 7094 raft_consensus.cc:2932] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 5a56fb218c4e482fac40da5c0be63a91 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: true } } peers { permanent_uuid: "5a56fb218c4e482fac40da5c0be63a91" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36639 } attrs { promote: true } } }
I20260926 03:31:58.470849 7225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:31:58.473733 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:58.452979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52462 (local address 127.0.97.1:41735)
0926 03:31:58.453567 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:58.453580 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:58.453603 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:31:58.453652 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:58.453658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:58.453899 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:58.454063 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:58.456019 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.457476 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.459007 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.459861 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.459988 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:58.465720 (+ 5732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:58.465733 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:58.465752 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:58.465797 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:58.469924 (+ 4127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:58.470707 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:58.470715 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:58.470719 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:58.470859 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:58.471309 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:58.471314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:58.471317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:58.471703 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:58.472929 (+ 1226us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:58.473077 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 03:31:58.473321 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":313,"threads_started":1}
I20260926 03:31:58.474273 7094 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:36639
I20260926 03:31:58.474607 7225 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [term 1 FOLLOWER]: Refusing update from remote peer aaf83112ebc946f5b71d630f6d3454b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 03:31:58.475143 6719 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:31:58.475198 7143 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [term 1 LEARNER]: Refusing update from remote peer aaf83112ebc946f5b71d630f6d3454b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 03:31:58.476203 7096 master.cc:537] Master@127.0.97.60:36639 shutting down...
I20260926 03:31:58.480933 7278 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 03:31:58.486158 7277 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a56fb218c4e482fac40da5c0be63a91" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36639 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:31:58.499194 7280 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "aaf83112ebc946f5b71d630f6d3454b0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: true } } peers { permanent_uuid: "5a56fb218c4e482fac40da5c0be63a91" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36639 } attrs { promote: true } } }
I20260926 03:31:58.499593 7280 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:58.500964 7143 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 5a56fb218c4e482fac40da5c0be63a91 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: true } } peers { permanent_uuid: "5a56fb218c4e482fac40da5c0be63a91" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36639 } attrs { promote: true } } }
I20260926 03:31:58.504407 7171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aaf83112ebc946f5b71d630f6d3454b0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: true } } peers { permanent_uuid: "5a56fb218c4e482fac40da5c0be63a91" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36639 } attrs { promote: true } } }
I20260926 03:31:58.504712 7171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [sys.catalog]: This master's current role is: LEARNER
I20260926 03:31:58.505261 7280 raft_consensus.cc:1050] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0: attempting to promote NON_VOTER 8692ab0d52964a07803500c4080166c4 to VOTER
I20260926 03:31:58.506021 7280 consensus_queue.cc:237] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: false } } peers { permanent_uuid: "5a56fb218c4e482fac40da5c0be63a91" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36639 } attrs { promote: true } }
I20260926 03:31:58.508037 7143 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [term 1 LEARNER]: Refusing update from remote peer aaf83112ebc946f5b71d630f6d3454b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 03:31:58.508739 7278 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 03:31:58.511021 7277 raft_consensus.cc:2932] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 8692ab0d52964a07803500c4080166c4 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: false } } peers { permanent_uuid: "5a56fb218c4e482fac40da5c0be63a91" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36639 } attrs { promote: true } } }
I20260926 03:31:58.515964 7096 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:31:58.515815 7142 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 8692ab0d52964a07803500c4080166c4 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: false } } peers { permanent_uuid: "5a56fb218c4e482fac40da5c0be63a91" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36639 } attrs { promote: true } } }
I20260926 03:31:58.516347 7096 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:31:58.516541 7096 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91: stopping tablet replica
I20260926 03:31:58.521148 7171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aaf83112ebc946f5b71d630f6d3454b0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: false } } peers { permanent_uuid: "5a56fb218c4e482fac40da5c0be63a91" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36639 } attrs { promote: true } } }
I20260926 03:31:58.521384 7171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:31:58.523996 7280 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aaf83112ebc946f5b71d630f6d3454b0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: false } } peers { permanent_uuid: "5a56fb218c4e482fac40da5c0be63a91" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36639 } attrs { promote: true } } }
I20260926 03:31:58.524281 7280 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:58.523890 7282 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "aaf83112ebc946f5b71d630f6d3454b0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: false } } peers { permanent_uuid: "5a56fb218c4e482fac40da5c0be63a91" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36639 } attrs { promote: true } } }
I20260926 03:31:58.524652 7282 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:58.525786 7096 master.cc:552] Master@127.0.97.60:36639 shutdown complete.
I20260926 03:31:58.525878 7096 master_runner.cc:305] Clearing existing system tablet
I20260926 03:31:58.526261 7096 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:58.526351 7096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:58.533206 7096 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.000s sys 0.003s
I20260926 03:31:58.537475 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:58.537046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36639 (local address 127.0.0.1:36138)
0926 03:31:58.537227 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:58.537314 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 03:31:58.538503 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:58.538178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36639 (local address 127.0.0.1:36150)
0926 03:31:58.538293 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:58.538364 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 03:31:58.538735 6521 consensus_peers.cc:487] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 -> Peer 5a56fb218c4e482fac40da5c0be63a91 (127.0.97.60:36639): Couldn't send request to peer 5a56fb218c4e482fac40da5c0be63a91. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36639: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 03:31:58.539137 7286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:58.539485 7096 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:58.539757 7096 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.004s
I20260926 03:31:58.539861 7096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-2/wal
uuid: "5a56fb218c4e482fac40da5c0be63a91"
format_stamp: "Formatted at 2026-09-26 03:31:58 on dist-test-slave-dt7p"
server_key: "55e7addc37616f6f7a794a31a3c34f9b"
server_key_iv: "7973e310ec5b297806b806e9a02c8f26"
server_key_version: "encryptionkey@0"
I20260926 03:31:58.544878 7096 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:31:58.552234 7096 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 03:31:58.552389 7096 log.cc:1196] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 03:31:58.552845 7096 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91: Deleting consensus metadata
I20260926 03:31:58.553102 7096 master_runner.cc:315] Copying system tablet from 127.0.97.62:46591
I20260926 03:31:58.555578 7096 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:46591
I20260926 03:31:58.571604 6860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:58.556419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54358 (local address 127.0.97.62:46591)
0926 03:31:58.556683 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:58.556690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:58.557239 (+ 549us) server_negotiation.cc:407] Connection header received
0926 03:31:58.557391 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:58.557397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:58.557475 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:58.557602 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:58.558933 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.560091 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.561946 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.562613 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.562733 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:58.565110 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:58.565116 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:58.565121 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:58.565157 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:58.568783 (+ 3626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:58.569526 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:58.569532 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:58.569536 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:58.569683 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:58.570109 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:58.570114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:58.570118 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:58.570455 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:58.570606 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:58.571036 (+ 430us) server_negotiation.cc:299] Negotiation successful
0926 03:31:58.571242 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260926 03:31:58.571686 7292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:58.556403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46591 (local address 127.0.0.1:54358)
0926 03:31:58.557097 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:58.557116 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:58.557263 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:58.557662 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:58.557667 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:58.558429 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:58.558739 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:58.558751 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.560248 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.560254 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:58.561781 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:58.561798 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:58.562771 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:58.562776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:58.562971 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:58.563725 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:58.563755 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:58.564844 (+ 1089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:58.568917 (+ 4073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:58.568923 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:58.568927 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:58.569385 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:58.569803 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:58.569809 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:58.569812 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:58.569984 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:58.570582 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:58.570587 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:58.570761 (+ 174us) client_negotiation.cc:769] Sending connection context
0926 03:31:58.570900 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 03:31:58.571237 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":391,"threads_started":1}
I20260926 03:31:58.572264 6555 tablet_copy_service.cc:140] P aaf83112ebc946f5b71d630f6d3454b0: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 5a56fb218c4e482fac40da5c0be63a91 ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:54358)
I20260926 03:31:58.572433 6555 tablet_copy_service.cc:161] P aaf83112ebc946f5b71d630f6d3454b0: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 5a56fb218c4e482fac40da5c0be63a91 at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:54358: session id = 5a56fb218c4e482fac40da5c0be63a91-00000000000000000000000000000000
I20260926 03:31:58.576018 6555 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 03:31:58.577924 7096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 03:31:58.585258 7096 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91: tablet copy: Starting download of 0 data blocks...
I20260926 03:31:58.585539 7096 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91: tablet copy: Starting download of 1 WAL segments...
I20260926 03:31:58.587895 7096 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 03:31:58.592144 6555 tablet_copy_service.cc:342] P aaf83112ebc946f5b71d630f6d3454b0: Request end of tablet copy session 5a56fb218c4e482fac40da5c0be63a91-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:54358
I20260926 03:31:58.592280 6555 tablet_copy_service.cc:433] P aaf83112ebc946f5b71d630f6d3454b0: ending tablet copy session 5a56fb218c4e482fac40da5c0be63a91-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 5a56fb218c4e482fac40da5c0be63a91
I20260926 03:31:58.603638 7096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:58.606662 7293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:58.606799 7294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:58.611186 7096 server_base.cc:1034] running on GCE node
W20260926 03:31:58.611331 7296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:58.611829 7096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:58.613560 7096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:58.613988 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 6596
I20260926 03:31:58.614810 7096 hybrid_clock.cc:648] HybridClock initialized: now 1790393518614771 us; error 28 us; skew 500 ppm
I20260926 03:31:58.615080 7096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:58 dist-test-slave-dt7p krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393518, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:58.617492 7096 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260926 03:31:58.624799 7096 webserver.cc:460] Webserver started at http://127.0.97.60:35307/ using document root <none> and password file <none>
I20260926 03:31:58.625169 7096 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:58.625274 7096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:58.629253 7096 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260926 03:31:58.631314 7303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:58.631803 7096 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260926 03:31:58.631928 7096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-2/wal
uuid: "5a56fb218c4e482fac40da5c0be63a91"
format_stamp: "Formatted at 2026-09-26 03:31:58 on dist-test-slave-dt7p"
server_key: "55e7addc37616f6f7a794a31a3c34f9b"
server_key_iv: "7973e310ec5b297806b806e9a02c8f26"
server_key_version: "encryptionkey@0"
I20260926 03:31:58.632179 7096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:58.640101 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:41735
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46591
--webserver_port=40049
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:46591,127.0.97.61:42461
--tserver_master_addrs=127.0.97.62:46591,127.0.97.61:42461,127.0.97.60:36639 with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:31:58.657015 7096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:58.661618 7096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:58.698547 7096 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:36639
I20260926 03:31:58.698606 7355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:36639 every 8 connection(s)
I20260926 03:31:58.699596 7096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/master-2/data/info.pb
I20260926 03:31:58.705874 7356 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:31:58.709503 7356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91: Bootstrap starting.
I20260926 03:31:58.728462 7356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:31:58.729000 7356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91: Bootstrap complete.
I20260926 03:31:58.730145 7356 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: false } } peers { permanent_uuid: "5a56fb218c4e482fac40da5c0be63a91" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36639 } attrs { promote: true } }
I20260926 03:31:58.730423 7356 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5a56fb218c4e482fac40da5c0be63a91, State: Initialized, Role: LEARNER
I20260926 03:31:58.730681 7356 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: false } } peers { permanent_uuid: "5a56fb218c4e482fac40da5c0be63a91" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36639 } attrs { promote: true } }
I20260926 03:31:58.731848 7356 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:31:58.731606 7359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: false } } peers { permanent_uuid: "5a56fb218c4e482fac40da5c0be63a91" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36639 } attrs { promote: true } } }
I20260926 03:31:58.732194 7359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [sys.catalog]: This master's current role is: LEARNER
I20260926 03:31:58.745945 7370 catalog_manager.cc:1259] Loaded cluster ID: 517c6d59af29429eb0c555bedc2befa8
I20260926 03:31:58.746068 7370 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91: loading cluster ID for follower catalog manager: success
I20260926 03:31:58.749513 7370 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91: acquiring CA information for follower catalog manager: success
I20260926 03:31:58.751796 7370 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 03:31:59.008313 7308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:59.008713 7308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:59.008862 7308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 03:31:59.010342 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:59.003595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36639 (local address 127.0.0.1:36160)
0926 03:31:59.003849 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:59.003870 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:59.003987 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:59.004856 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:59.004862 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:59.004912 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:31:59.005115 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:59.005126 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:59.006968 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:59.006976 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:59.008187 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:59.008200 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:59.009795 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:59.009800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:59.009924 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:59.009949 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 03:31:59.010029 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 03:31:59.010161 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260926 03:31:59.010650 7374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:59.003743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36160 (local address 127.0.97.60:36639)
0926 03:31:59.004482 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:59.004488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:59.004507 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:31:59.004601 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:59.004606 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:59.004689 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:59.004803 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:31:59.005309 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:59.006807 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:59.008360 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:59.009622 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:59.009755 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:59.009957 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:59.010049 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 03:31:59.010174 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":461,"threads_started":1}
I20260926 03:31:59.011485 7331 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 03:31:59.012032 7359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aaf83112ebc946f5b71d630f6d3454b0. Latest consensus state: current_term: 1 leader_uuid: "aaf83112ebc946f5b71d630f6d3454b0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: false } } peers { permanent_uuid: "5a56fb218c4e482fac40da5c0be63a91" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36639 } attrs { promote: true } } }
I20260926 03:31:59.012295 7359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [sys.catalog]: This master's current role is: LEARNER
W20260926 03:31:59.017637 7308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:59.017761 7308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:59.017966 7308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:31:59.029035 7308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:41735
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=40049
--tserver_master_addrs=127.0.97.62:46591,127.0.97.61:42461,127.0.97.60:36639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:59.030401 7308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:59.031796 7308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:59.046008 7379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:59.045964 7377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:59.046595 7376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:59.047276 7308 server_base.cc:1034] running on GCE node
I20260926 03:31:59.048004 7308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:59.049091 7308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:59.050297 7308 hybrid_clock.cc:648] HybridClock initialized: now 1790393519050262 us; error 31 us; skew 500 ppm
I20260926 03:31:59.050679 7308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:59 dist-test-slave-dt7p krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393519, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:59.054056 7308 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260926 03:31:59.055437 7308 webserver.cc:460] Webserver started at http://127.0.97.1:40049/ using document root <none> and password file <none>
I20260926 03:31:59.056074 7308 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:59.056186 7308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:59.061600 7308 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:31:59.064148 7386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:59.065558 7308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 03:31:59.065738 7308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-0/wal
uuid: "6b0027955ae14788b8afa531cbfd18e1"
format_stamp: "Formatted at 2026-09-26 03:31:56 on dist-test-slave-dt7p"
server_key: "5624f1180d45ec01fddbb19db1605f99"
server_key_iv: "e813559505b187586949b15e5989fa78"
server_key_version: "encryptionkey@0"
I20260926 03:31:59.066673 7308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:59.082592 7308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:59.086716 7308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:59.087010 7308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:59.087927 7308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:31:59.089406 7308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:31:59.089500 7308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:59.089649 7308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:31:59.089743 7308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:59.124944 7308 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41735
I20260926 03:31:59.125007 7498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41735 every 8 connection(s)
I20260926 03:31:59.126444 7308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:31:59.136400 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 7308
I20260926 03:31:59.136891 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 6730
Sep 26 03:31:59 dist-test-slave-dt7p krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393519, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 26 03:31:59 dist-test-slave-dt7p krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393519, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 26 03:31:59 dist-test-slave-dt7p krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393519, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:31:59.158731 7374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:59.133404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45727 (local address 127.0.97.60:36639)
0926 03:31:59.133565 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:59.133574 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:59.134014 (+ 440us) server_negotiation.cc:407] Connection header received
0926 03:31:59.134566 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:59.134572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:59.134657 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:59.134781 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:59.137441 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:59.138894 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:59.142111 (+ 3217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:59.143074 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:59.143215 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:59.150486 (+ 7271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:59.150493 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:59.150497 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:59.150539 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:59.155398 (+ 4859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:59.156290 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:59.156295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:59.156300 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:59.156450 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:59.156975 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:59.156980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:59.156984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:59.157366 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:59.157499 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:59.158141 (+ 642us) server_negotiation.cc:299] Negotiation successful
0926 03:31:59.158376 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 03:31:59.159787 7503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:59.129791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49903 (local address 127.0.97.62:46591)
0926 03:31:59.130290 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:59.130297 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:59.131604 (+ 1307us) server_negotiation.cc:407] Connection header received
0926 03:31:59.134588 (+ 2984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:59.134597 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:59.134677 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:59.134755 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:59.137376 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:59.138911 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:59.147115 (+ 8204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:59.147807 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:59.147931 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:59.153006 (+ 5075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:59.153015 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:59.153021 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:59.153068 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:59.156520 (+ 3452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:59.157262 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:59.157267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:59.157270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:59.157396 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:59.157851 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:59.157855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:59.157858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:59.158164 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:59.158325 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:59.159111 (+ 786us) server_negotiation.cc:299] Negotiation successful
0926 03:31:59.159376 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":256,"threads_started":1}
I20260926 03:31:59.159950 7506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:59.130288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42461 (local address 127.0.97.1:38895)
0926 03:31:59.133056 (+ 2768us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:59.133073 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:59.134294 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:59.135063 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:59.135072 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:59.136048 (+ 976us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:59.137051 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:59.137076 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:59.139071 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:59.139075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:59.141891 (+ 2816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:59.141911 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:59.143521 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:59.143530 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:59.143930 (+ 400us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:59.147220 (+ 3290us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:59.147241 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:59.150207 (+ 2966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:59.155179 (+ 4972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:59.155190 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:59.155208 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:59.155840 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:59.156276 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:59.156282 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:59.156286 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:59.156493 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:59.157252 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:59.157259 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:59.157667 (+ 408us) client_negotiation.cc:769] Sending connection context
0926 03:31:59.157996 (+ 329us) client_negotiation.cc:240] Negotiation successful
0926 03:31:59.158488 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2618,"mutex_wait_us":243,"spinlock_wait_cycles":27008,"thread_start_us":545,"threads_started":1}
I20260926 03:31:59.159994 7507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:59.133220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36639 (local address 127.0.97.1:45727)
0926 03:31:59.133892 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:59.133909 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:59.134306 (+ 397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:59.135061 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:59.135070 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:59.136058 (+ 988us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:59.137253 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:59.137268 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:59.139183 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:59.139192 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:59.141896 (+ 2704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:59.141916 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:59.143627 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:59.143637 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:59.143936 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:59.145222 (+ 1286us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:59.145245 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:59.150212 (+ 4967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:59.155555 (+ 5343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:59.155560 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:59.155564 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:59.156150 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:59.156637 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:59.156642 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:59.156646 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:59.156842 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:59.157584 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:59.157589 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:59.157812 (+ 223us) client_negotiation.cc:769] Sending connection context
0926 03:31:59.159067 (+ 1255us) client_negotiation.cc:240] Negotiation successful
0926 03:31:59.159369 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":99,"spinlock_wait_cycles":9984,"thread_start_us":345,"threads_started":1}
I20260926 03:31:59.161243 7504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:59.129830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38895 (local address 127.0.97.61:42461)
0926 03:31:59.133865 (+ 4035us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:59.133872 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:59.133890 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:31:59.134652 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:59.134657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:59.134748 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:59.134839 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:59.137276 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:59.138859 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:59.142121 (+ 3262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:59.142992 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:59.143143 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:59.150461 (+ 7318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:59.150468 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:59.150473 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:59.150513 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:59.155019 (+ 4506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:59.156000 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:59.156006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:59.156010 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:59.156145 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:59.156650 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:59.156655 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:59.156660 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:59.157030 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:59.157186 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:59.160669 (+ 3483us) server_negotiation.cc:299] Negotiation successful
0926 03:31:59.160873 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3925,"thread_start_us":3888,"threads_started":1}
I20260926 03:31:59.161274 7505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:59.130287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46591 (local address 127.0.97.1:49903)
0926 03:31:59.131417 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:59.131479 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:59.134294 (+ 2815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:59.135063 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:59.135072 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:59.136048 (+ 976us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:59.137211 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:59.137227 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:59.144771 (+ 7544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:59.144779 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:59.146954 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:59.146984 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:59.148759 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:59.148764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:59.148917 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:59.149532 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:59.149554 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:59.152786 (+ 3232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:59.156705 (+ 3919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:59.156711 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:59.156715 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:59.157134 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:59.157540 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:59.157544 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:59.157547 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:59.157713 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:59.158434 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:59.158440 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:59.158579 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 03:31:59.160624 (+ 2045us) client_negotiation.cc:240] Negotiation successful
0926 03:31:59.160874 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"mutex_wait_us":122,"spinlock_wait_cycles":47360,"thread_start_us":2331,"threads_started":1}
I20260926 03:31:59.161970 7500 heartbeater.cc:347] Connected to a master server at 127.0.97.60:36639
I20260926 03:31:59.162531 7501 heartbeater.cc:347] Connected to a master server at 127.0.97.61:42461
I20260926 03:31:59.162724 7501 heartbeater.cc:464] Registering TS with master...
I20260926 03:31:59.162729 7500 heartbeater.cc:464] Registering TS with master...
I20260926 03:31:59.163686 7500 heartbeater.cc:511] Master 127.0.97.60:36639 requested a full tablet report, sending...
I20260926 03:31:59.163664 7501 heartbeater.cc:511] Master 127.0.97.61:42461 requested a full tablet report, sending...
I20260926 03:31:59.163868 7499 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46591
I20260926 03:31:59.164002 7499 heartbeater.cc:464] Registering TS with master...
I20260926 03:31:59.164539 7499 heartbeater.cc:511] Master 127.0.97.62:46591 requested a full tablet report, sending...
Sep 26 03:31:59 dist-test-slave-dt7p krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260926 03:31:59.166311 7321 ts_manager.cc:194] Registered new tserver with Master: 6b0027955ae14788b8afa531cbfd18e1 (127.0.97.1:41735)
I20260926 03:31:59.166296 7133 ts_manager.cc:194] Registered new tserver with Master: 6b0027955ae14788b8afa531cbfd18e1 (127.0.97.1:41735)
I20260926 03:31:59.167482 6535 ts_manager.cc:194] Re-registered known tserver with Master: 6b0027955ae14788b8afa531cbfd18e1 (127.0.97.1:41735)
I20260926 03:31:59.168478 6535 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49903
I20260926 03:31:59.173748 7509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:59.139449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41510 (local address 127.0.97.2:34011)
0926 03:31:59.149007 (+ 9558us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:59.149017 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:59.150673 (+ 1656us) server_negotiation.cc:407] Connection header received
0926 03:31:59.151155 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:59.151166 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:59.151454 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:59.151556 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:59.154131 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:59.155846 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:59.157913 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:59.158869 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:59.158989 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:59.167050 (+ 8061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:59.167064 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:59.167078 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:59.167115 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:59.170685 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:59.171521 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:59.171526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:59.171528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:59.171651 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:59.171985 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:59.171989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:59.171992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:59.172284 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:59.172551 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:59.172907 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 03:31:59.173088 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9369,"thread_start_us":9661,"threads_started":1}
I20260926 03:31:59.174908 6853 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:31:59.261233 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 6730
I20260926 03:31:59.277407 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:34011
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46591
--webserver_port=35415
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:46591,127.0.97.61:42461
--tserver_master_addrs=127.0.97.62:46591,127.0.97.61:42461,127.0.97.60:36639 with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:31:59.404232 7277 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aaf83112ebc946f5b71d630f6d3454b0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: false } } peers { permanent_uuid: "5a56fb218c4e482fac40da5c0be63a91" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36639 } attrs { promote: true } } }
I20260926 03:31:59.404675 7277 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:59.406070 7277 raft_consensus.cc:1050] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0: attempting to promote NON_VOTER 5a56fb218c4e482fac40da5c0be63a91 to VOTER
I20260926 03:31:59.406991 7277 consensus_queue.cc:237] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: false } } peers { permanent_uuid: "5a56fb218c4e482fac40da5c0be63a91" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36639 } attrs { promote: false } }
I20260926 03:31:59.409065 7143 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [term 1 FOLLOWER]: Refusing update from remote peer aaf83112ebc946f5b71d630f6d3454b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 03:31:59.409070 7331 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [term 1 LEARNER]: Refusing update from remote peer aaf83112ebc946f5b71d630f6d3454b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 03:31:59.409720 7513 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 03:31:59.410154 7277 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a56fb218c4e482fac40da5c0be63a91" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36639 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 03:31:59.412714 7513 raft_consensus.cc:2932] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5a56fb218c4e482fac40da5c0be63a91 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: false } } peers { permanent_uuid: "5a56fb218c4e482fac40da5c0be63a91" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36639 } attrs { promote: false } } }
I20260926 03:31:59.413416 7143 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5a56fb218c4e482fac40da5c0be63a91 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: false } } peers { permanent_uuid: "5a56fb218c4e482fac40da5c0be63a91" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36639 } attrs { promote: false } } }
I20260926 03:31:59.416433 7513 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "aaf83112ebc946f5b71d630f6d3454b0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: false } } peers { permanent_uuid: "5a56fb218c4e482fac40da5c0be63a91" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36639 } attrs { promote: false } } }
I20260926 03:31:59.416738 7513 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:59.416611 7512 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aaf83112ebc946f5b71d630f6d3454b0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: false } } peers { permanent_uuid: "5a56fb218c4e482fac40da5c0be63a91" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36639 } attrs { promote: false } } }
I20260926 03:31:59.416950 7512 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:59.417053 7513 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aaf83112ebc946f5b71d630f6d3454b0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: false } } peers { permanent_uuid: "5a56fb218c4e482fac40da5c0be63a91" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36639 } attrs { promote: false } } }
I20260926 03:31:59.417232 7513 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:31:59.416725 7331 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5a56fb218c4e482fac40da5c0be63a91 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: false } } peers { permanent_uuid: "5a56fb218c4e482fac40da5c0be63a91" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36639 } attrs { promote: false } } }
I20260926 03:31:59.418530 7516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aaf83112ebc946f5b71d630f6d3454b0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: false } } peers { permanent_uuid: "5a56fb218c4e482fac40da5c0be63a91" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36639 } attrs { promote: false } } }
I20260926 03:31:59.419005 7516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:31:59.422063 7359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aaf83112ebc946f5b71d630f6d3454b0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: false } } peers { permanent_uuid: "5a56fb218c4e482fac40da5c0be63a91" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36639 } attrs { promote: false } } }
I20260926 03:31:59.422312 7359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 03:31:59.615947 7511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:31:59.616235 7511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:31:59.616397 7511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:31:59.623719 7511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:31:59.623826 7511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:31:59.624032 7511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:31:59.634230 7511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:34011
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=35415
--tserver_master_addrs=127.0.97.62:46591,127.0.97.61:42461,127.0.97.60:36639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:31:59.635710 7511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:31:59.637161 7511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:31:59.651150 7524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:59.651122 7521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:31:59.651181 7522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:31:59.651885 7511 server_base.cc:1034] running on GCE node
I20260926 03:31:59.652530 7511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:31:59.653632 7511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:31:59.654838 7511 hybrid_clock.cc:648] HybridClock initialized: now 1790393519654779 us; error 50 us; skew 500 ppm
I20260926 03:31:59.655114 7511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:31:59 dist-test-slave-dt7p krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393519, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:31:59.658486 7511 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260926 03:31:59.659992 7511 webserver.cc:460] Webserver started at http://127.0.97.2:35415/ using document root <none> and password file <none>
I20260926 03:31:59.660792 7511 fs_manager.cc:359] Metadata directory not provided
I20260926 03:31:59.660974 7511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:31:59.666543 7511 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 03:31:59.669227 7531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:59.670607 7511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:31:59.670773 7511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-1/wal
uuid: "b61e6b996c96424aa56ab30a31376891"
format_stamp: "Formatted at 2026-09-26 03:31:56 on dist-test-slave-dt7p"
server_key: "92b1bfa2587c0f6d71199fab9a624db7"
server_key_iv: "e2b931e3e93ccfe666fafd923287e6a4"
server_key_version: "encryptionkey@0"
I20260926 03:31:59.671813 7511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:31:59.688449 7511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:31:59.692945 7511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:31:59.693230 7511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:31:59.694200 7511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:31:59.695552 7511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:31:59.695638 7511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:59.695806 7511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:31:59.695904 7511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:31:59.735376 7511 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34011
I20260926 03:31:59.735497 7643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34011 every 8 connection(s)
I20260926 03:31:59.736923 7511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:31:59.737828 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 7511
I20260926 03:31:59.738130 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 6864
Sep 26 03:31:59 dist-test-slave-dt7p krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393519, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 26 03:31:59 dist-test-slave-dt7p krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393519, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 26 03:31:59 dist-test-slave-dt7p krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393519, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 26 03:31:59 dist-test-slave-dt7p krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260926 03:31:59.792694 7654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:59.750575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43693 (local address 127.0.97.62:46591)
0926 03:31:59.752927 (+ 2352us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:59.752936 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:59.772620 (+ 19684us) server_negotiation.cc:407] Connection header received
0926 03:31:59.772695 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:59.772700 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:59.772778 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:59.772899 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:59.774569 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:59.776137 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:59.777937 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:59.778521 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:59.778607 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:59.785022 (+ 6415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:59.785027 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:59.785032 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:59.785069 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:59.788853 (+ 3784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:59.789839 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:59.789845 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:59.789849 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:59.789982 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:59.790530 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:59.790534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:59.790538 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:59.790860 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:59.790991 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:59.792066 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0926 03:31:59.792301 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2195,"thread_start_us":2103,"threads_started":1}
I20260926 03:31:59.793879 7653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:59.740541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42461 (local address 127.0.97.2:52347)
0926 03:31:59.749787 (+ 9246us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:59.749837 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:59.756400 (+ 6563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:59.769329 (+ 12929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:59.769334 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:59.769866 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:59.770139 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:59.770150 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:59.773626 (+ 3476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:59.773632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:59.775760 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:59.775772 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:59.779082 (+ 3310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:59.779088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:59.779255 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:59.780009 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:59.780036 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:59.783918 (+ 3882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:59.788899 (+ 4981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:59.788911 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:59.788932 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:59.789704 (+ 772us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:59.790112 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:59.790117 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:59.790120 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:59.790401 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:59.791077 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:59.791085 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:59.791605 (+ 520us) client_negotiation.cc:769] Sending connection context
0926 03:31:59.791954 (+ 349us) client_negotiation.cc:240] Negotiation successful
0926 03:31:59.792455 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8914,"mutex_wait_us":84,"spinlock_wait_cycles":58368,"thread_start_us":27703,"threads_started":1}
I20260926 03:31:59.793901 7651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:59.754375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46591 (local address 127.0.97.2:43693)
0926 03:31:59.768187 (+ 13812us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:59.768208 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:59.768325 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:59.772990 (+ 4665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:59.772998 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:59.773849 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:59.774338 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:59.774354 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:59.776336 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:59.776339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:59.777789 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:59.777802 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:59.778894 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:59.778900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:59.779190 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:59.781683 (+ 2493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:59.781707 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:59.784765 (+ 3058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:59.788985 (+ 4220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:59.788991 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:59.788999 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:59.789704 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:59.790123 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:59.790127 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:59.790130 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:59.790407 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:59.791076 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:59.791085 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:59.791599 (+ 514us) client_negotiation.cc:769] Sending connection context
0926 03:31:59.793048 (+ 1449us) client_negotiation.cc:240] Negotiation successful
0926 03:31:59.793305 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13678,"mutex_wait_us":76,"spinlock_wait_cycles":22144,"thread_start_us":637,"threads_started":1}
I20260926 03:31:59.795146 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:59.740800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54583 (local address 127.0.97.60:36639)
0926 03:31:59.756199 (+ 15399us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:59.756208 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:59.756235 (+ 27us) server_negotiation.cc:407] Connection header received
0926 03:31:59.756947 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:59.756956 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:59.757055 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:59.757170 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:59.760124 (+ 2954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:59.761586 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:59.774672 (+ 13086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:59.775763 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:59.775902 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:59.786144 (+ 10242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:59.786151 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:59.786155 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:59.786197 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:59.790451 (+ 4254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:59.791322 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:59.791327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:59.791331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:59.791475 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:59.791923 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:59.791929 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:59.791932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:59.792282 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:59.792427 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:59.793949 (+ 1522us) server_negotiation.cc:299] Negotiation successful
0926 03:31:59.794669 (+ 720us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15245,"thread_start_us":15445,"threads_started":1}
I20260926 03:31:59.795385 7650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:59.739930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52347 (local address 127.0.97.61:42461)
0926 03:31:59.768927 (+ 28997us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:59.768978 (+ 51us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:59.769000 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:31:59.769062 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:59.769069 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:59.769154 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:59.770286 (+ 1132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:59.770425 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:59.771561 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:59.775956 (+ 4395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:59.778740 (+ 2784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:59.778853 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:59.784798 (+ 5945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:59.784809 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:59.784818 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:59.784908 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:59.788728 (+ 3820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:59.789839 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:59.789845 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:59.789848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:59.789982 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:59.790529 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:59.790534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:59.790538 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:59.790878 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:59.790991 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:59.794374 (+ 3383us) server_negotiation.cc:299] Negotiation successful
0926 03:31:59.794530 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28839,"thread_start_us":38322,"threads_started":1}
I20260926 03:31:59.795455 7655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:59.740541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36639 (local address 127.0.97.2:54583)
0926 03:31:59.754983 (+ 14442us) negotiation.cc:107] Waiting for socket to connect
0926 03:31:59.755006 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:31:59.756597 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:31:59.757705 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:31:59.757722 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:31:59.758864 (+ 1142us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:31:59.759878 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:59.759907 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:59.761890 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:59.761898 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:59.774472 (+ 12574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:31:59.774487 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:31:59.781862 (+ 7375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:31:59.781869 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:31:59.782070 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:59.782750 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:31:59.782776 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:31:59.785411 (+ 2635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:31:59.790619 (+ 5208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:59.790626 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:59.790631 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:59.791176 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:59.791628 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:31:59.791635 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:31:59.791640 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:31:59.791791 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:31:59.792489 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:31:59.792494 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:31:59.792679 (+ 185us) client_negotiation.cc:769] Sending connection context
0926 03:31:59.792807 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 03:31:59.793103 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14237,"mutex_wait_us":425,"thread_start_us":8295,"threads_started":1}
I20260926 03:31:59.795524 7644 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46591
I20260926 03:31:59.796120 7644 heartbeater.cc:464] Registering TS with master...
I20260926 03:31:59.797510 7645 heartbeater.cc:347] Connected to a master server at 127.0.97.60:36639
I20260926 03:31:59.797506 7646 heartbeater.cc:347] Connected to a master server at 127.0.97.61:42461
I20260926 03:31:59.797662 7645 heartbeater.cc:464] Registering TS with master...
I20260926 03:31:59.797734 7646 heartbeater.cc:464] Registering TS with master...
I20260926 03:31:59.797735 7644 heartbeater.cc:511] Master 127.0.97.62:46591 requested a full tablet report, sending...
I20260926 03:31:59.798046 7645 heartbeater.cc:511] Master 127.0.97.60:36639 requested a full tablet report, sending...
I20260926 03:31:59.798238 7646 heartbeater.cc:511] Master 127.0.97.61:42461 requested a full tablet report, sending...
I20260926 03:31:59.799486 7321 ts_manager.cc:194] Registered new tserver with Master: b61e6b996c96424aa56ab30a31376891 (127.0.97.2:34011)
I20260926 03:31:59.799528 7133 ts_manager.cc:194] Registered new tserver with Master: b61e6b996c96424aa56ab30a31376891 (127.0.97.2:34011)
I20260926 03:31:59.799612 6535 ts_manager.cc:194] Re-registered known tserver with Master: b61e6b996c96424aa56ab30a31376891 (127.0.97.2:34011)
I20260926 03:31:59.800786 6535 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43693
I20260926 03:31:59.803184 7649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:31:59.739763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53746 (local address 127.0.97.3:36319)
0926 03:31:59.772797 (+ 33034us) server_negotiation.cc:206] Beginning negotiation
0926 03:31:59.772807 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:31:59.772840 (+ 33us) server_negotiation.cc:407] Connection header received
0926 03:31:59.772912 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:31:59.772921 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:31:59.773212 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:31:59.773380 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:31:59.780217 (+ 6837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:59.782845 (+ 2628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:59.784812 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:31:59.785768 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:31:59.785900 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:31:59.792367 (+ 6467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:31:59.792382 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:31:59.792401 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:31:59.792452 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:31:59.798868 (+ 6416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:59.801027 (+ 2159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:59.801035 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:59.801039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:59.801194 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:31:59.801630 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:31:59.801635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:31:59.801638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:31:59.801956 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:31:59.802244 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0926 03:31:59.802599 (+ 355us) server_negotiation.cc:299] Negotiation successful
0926 03:31:59.802844 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32867,"thread_start_us":32693,"threads_started":1}
I20260926 03:31:59.804255 6987 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:31:59.888411 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 6864
I20260926 03:31:59.906754 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:36319
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46591
--webserver_port=37229
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:46591,127.0.97.61:42461
--tserver_master_addrs=127.0.97.62:46591,127.0.97.61:42461,127.0.97.60:36639 with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:32:00.170895 7499 heartbeater.cc:503] Master 127.0.97.62:46591 was elected leader, sending a full tablet report...
W20260926 03:32:00.253551 7657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:32:00.253846 7657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:32:00.254029 7657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:32:00.261430 7657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:32:00.261530 7657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:32:00.261641 7657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:32:00.271217 7657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:36319
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=37229
--tserver_master_addrs=127.0.97.62:46591,127.0.97.61:42461,127.0.97.60:36639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:32:00.272452 7657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:32:00.273685 7657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:32:00.287606 7662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:00.287640 7663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:00.287518 7665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:32:00.288373 7657 server_base.cc:1034] running on GCE node
I20260926 03:32:00.289036 7657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:32:00.289906 7657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:32:00.291115 7657 hybrid_clock.cc:648] HybridClock initialized: now 1790393520291056 us; error 55 us; skew 500 ppm
I20260926 03:32:00.291446 7657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:32:00 dist-test-slave-dt7p krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393520, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:32:00.294660 7657 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260926 03:32:00.296183 7657 webserver.cc:460] Webserver started at http://127.0.97.3:37229/ using document root <none> and password file <none>
I20260926 03:32:00.296923 7657 fs_manager.cc:359] Metadata directory not provided
I20260926 03:32:00.297035 7657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:32:00.302269 7657 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 03:32:00.305162 7672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:00.306568 7657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:32:00.306772 7657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-2/wal
uuid: "91c53a144b0242e491f9e8b8aaef3f02"
format_stamp: "Formatted at 2026-09-26 03:31:57 on dist-test-slave-dt7p"
server_key: "38190125c2b2bb94e02b6ec37d5d707d"
server_key_iv: "ec38cc02266287a140de0d0ac165e39a"
server_key_version: "encryptionkey@0"
I20260926 03:32:00.307741 7657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:32:00.318938 7657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:32:00.324780 7657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:32:00.325148 7657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:32:00.326254 7657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:32:00.327776 7657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:32:00.327898 7657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:00.328003 7657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:32:00.328085 7657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:00.364363 7657 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36319
I20260926 03:32:00.364460 7784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36319 every 8 connection(s)
I20260926 03:32:00.365936 7657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:32:00.366478 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 7657
Sep 26 03:32:00 dist-test-slave-dt7p krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393520, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 26 03:32:00 dist-test-slave-dt7p krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393520, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 26 03:32:00 dist-test-slave-dt7p krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393520, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:32:00.418802 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:00.369089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38969 (local address 127.0.97.62:46591)
0926 03:32:00.389396 (+ 20307us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:00.389402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:00.389421 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:32:00.389494 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:00.389498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:00.389570 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:00.389688 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:00.391263 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.392295 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.397226 (+ 4931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.397945 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.398088 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:00.409453 (+ 11365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:00.409459 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:00.409463 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:00.409501 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:00.413820 (+ 4319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:00.416150 (+ 2330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:00.416158 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:00.416163 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:00.416314 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:00.416905 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:00.416911 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:00.416916 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:00.417310 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:00.417460 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:00.418163 (+ 703us) server_negotiation.cc:299] Negotiation successful
0926 03:32:00.418376 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20178,"thread_start_us":20700,"threads_started":1}
I20260926 03:32:00.419090 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:00.378608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40561 (local address 127.0.97.60:36639)
0926 03:32:00.388360 (+ 9752us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:00.388366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:00.388463 (+ 97us) server_negotiation.cc:407] Connection header received
0926 03:32:00.388815 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:00.388821 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:00.388898 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:00.389002 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:00.395180 (+ 6178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.396131 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.400360 (+ 4229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.401180 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.401293 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:00.408503 (+ 7210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:00.408510 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:00.408514 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:00.408557 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:00.412908 (+ 4351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:00.413911 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:00.413919 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:00.413922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:00.414053 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:00.414588 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:00.414594 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:00.414598 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:00.414964 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:00.415084 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:00.418563 (+ 3479us) server_negotiation.cc:299] Negotiation successful
0926 03:32:00.418769 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9633,"thread_start_us":9784,"threads_started":1}
I20260926 03:32:00.419391 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:00.369890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46591 (local address 127.0.97.3:38969)
0926 03:32:00.387838 (+ 17948us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:00.387859 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:00.387966 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:00.389747 (+ 1781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:00.389751 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:00.390513 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:00.390786 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:00.390798 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.392425 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.392431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:00.394042 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:00.394054 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.402761 (+ 8707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.402770 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:00.403007 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:00.405326 (+ 2319us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:00.405355 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:00.408263 (+ 2908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:32:00.414449 (+ 6186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:32:00.414454 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:32:00.414457 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:32:00.414948 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:32:00.416436 (+ 1488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:32:00.416441 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:32:00.416445 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:32:00.416602 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:32:00.417453 (+ 851us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:32:00.417457 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:32:00.417601 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 03:32:00.417727 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 03:32:00.418006 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17780,"mutex_wait_us":393,"thread_start_us":6856,"threads_started":1}
I20260926 03:32:00.420043 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:00.378531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36639 (local address 127.0.97.3:40561)
0926 03:32:00.388325 (+ 9794us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:00.388347 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:00.388607 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:00.389969 (+ 1362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:00.389974 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:00.393503 (+ 3529us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:00.393827 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:00.393837 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.398351 (+ 4514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.398359 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:00.400177 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:00.400194 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.401627 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.401635 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:00.401868 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:00.403478 (+ 1610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:00.403509 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:00.407367 (+ 3858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:32:00.413080 (+ 5713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:32:00.413091 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:32:00.413109 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:32:00.413767 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:32:00.414205 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:32:00.414210 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:32:00.414214 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:32:00.414449 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:32:00.415163 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:32:00.415172 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:32:00.415621 (+ 449us) client_negotiation.cc:769] Sending connection context
0926 03:32:00.415899 (+ 278us) client_negotiation.cc:240] Negotiation successful
0926 03:32:00.417909 (+ 2010us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9646,"thread_start_us":6281,"threads_started":1}
I20260926 03:32:00.421044 7785 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46591
I20260926 03:32:00.421355 7785 heartbeater.cc:464] Registering TS with master...
I20260926 03:32:00.422533 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:00.369890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42461 (local address 127.0.97.3:38353)
0926 03:32:00.377488 (+ 7598us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:00.377546 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:00.380456 (+ 2910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:00.385384 (+ 4928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:00.385404 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:00.386476 (+ 1072us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:00.387529 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:00.387559 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.389164 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.389168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:00.391060 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:00.391069 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.392619 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.392627 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:00.392929 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:00.394078 (+ 1149us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:00.394120 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:00.397010 (+ 2890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:32:00.413874 (+ 16864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:32:00.413880 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:32:00.413884 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:32:00.414318 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:32:00.420681 (+ 6363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:32:00.420686 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:32:00.420690 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:32:00.420820 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:32:00.421410 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:32:00.421414 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:32:00.421579 (+ 165us) client_negotiation.cc:769] Sending connection context
0926 03:32:00.421730 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 03:32:00.422060 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7332,"thread_start_us":18446,"threads_started":1}
I20260926 03:32:00.422272 7785 heartbeater.cc:511] Master 127.0.97.62:46591 requested a full tablet report, sending...
I20260926 03:32:00.422721 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:00.370230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38353 (local address 127.0.97.61:42461)
0926 03:32:00.377008 (+ 6778us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:00.377013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:00.377695 (+ 682us) server_negotiation.cc:407] Connection header received
0926 03:32:00.384732 (+ 7037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:00.384741 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:00.384861 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:00.385059 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:00.387718 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.388891 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.391190 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.391833 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.391960 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:00.402252 (+ 10292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:00.402262 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:00.402270 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:00.402321 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:00.413755 (+ 11434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:00.420391 (+ 6636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:00.420396 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:00.420401 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:00.420553 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:00.420955 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:00.420960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:00.420964 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:00.421300 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:00.421433 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:00.422153 (+ 720us) server_negotiation.cc:299] Negotiation successful
0926 03:32:00.422374 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6657,"thread_start_us":6525,"threads_started":1}
I20260926 03:32:00.423439 7787 heartbeater.cc:347] Connected to a master server at 127.0.97.61:42461
I20260926 03:32:00.423559 7787 heartbeater.cc:464] Registering TS with master...
I20260926 03:32:00.424010 7787 heartbeater.cc:511] Master 127.0.97.61:42461 requested a full tablet report, sending...
I20260926 03:32:00.424091 7786 heartbeater.cc:347] Connected to a master server at 127.0.97.60:36639
I20260926 03:32:00.424186 7786 heartbeater.cc:464] Registering TS with master...
I20260926 03:32:00.424626 7786 heartbeater.cc:511] Master 127.0.97.60:36639 requested a full tablet report, sending...
I20260926 03:32:00.425042 6535 ts_manager.cc:194] Re-registered known tserver with Master: 91c53a144b0242e491f9e8b8aaef3f02 (127.0.97.3:36319)
I20260926 03:32:00.425796 7321 ts_manager.cc:194] Registered new tserver with Master: 91c53a144b0242e491f9e8b8aaef3f02 (127.0.97.3:36319)
I20260926 03:32:00.426146 6535 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38969
I20260926 03:32:00.427016 7133 ts_manager.cc:194] Registered new tserver with Master: 91c53a144b0242e491f9e8b8aaef3f02 (127.0.97.3:36319)
I20260926 03:32:00.436630 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:00.397412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44214 (local address 127.0.97.61:42461)
0926 03:32:00.408483 (+ 11071us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:00.408492 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:00.408516 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:32:00.408608 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:00.408617 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:00.408718 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:00.408850 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:00.415206 (+ 6356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.416350 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.423229 (+ 6879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.423898 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.424018 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:00.429635 (+ 5617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:00.429640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:00.429643 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:00.429691 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:00.433546 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:00.434365 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:00.434371 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:00.434376 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:00.434522 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:00.434947 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:00.434952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:00.434955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:00.435283 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:00.435547 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:00.435913 (+ 366us) server_negotiation.cc:299] Negotiation successful
0926 03:32:00.436200 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10905,"thread_start_us":10768,"threads_started":1}
I20260926 03:32:00.438715 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:00.403190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36170 (local address 127.0.97.60:36639)
0926 03:32:00.411602 (+ 8412us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:00.411608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:00.411631 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:32:00.411681 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:00.411686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:00.411768 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:00.411879 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:00.416889 (+ 5010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.420017 (+ 3128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.427639 (+ 7622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.428034 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.428168 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:00.431716 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:00.431721 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:00.431724 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:00.431753 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:00.435338 (+ 3585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:00.436030 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:00.436035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:00.436039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:00.436207 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:00.436682 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:00.436686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:00.436692 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:00.437017 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:00.437269 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:00.438192 (+ 923us) server_negotiation.cc:299] Negotiation successful
0926 03:32:00.438397 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8277,"thread_start_us":8121,"threads_started":1}
I20260926 03:32:00.438727 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:00.390611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54360 (local address 127.0.97.62:46591)
0926 03:32:00.412539 (+ 21928us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:00.412544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:00.412600 (+ 56us) server_negotiation.cc:407] Connection header received
0926 03:32:00.412659 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:00.412663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:00.412730 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:00.412842 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:00.416763 (+ 3921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.420234 (+ 3471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.423119 (+ 2885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.423898 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.424018 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:00.430624 (+ 6606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:00.430630 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:00.430635 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:00.430673 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:00.434235 (+ 3562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:00.435608 (+ 1373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:00.435611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:00.435614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:00.435719 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:00.437138 (+ 1419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:00.437143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:00.437146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:00.437539 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:00.437761 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:00.438110 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 03:32:00.438331 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21783,"thread_start_us":25024,"threads_started":1}
I20260926 03:32:00.442665 6535 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:32:00.450151 6535 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:32:00.483011 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:00.471136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41514 (local address 127.0.97.2:34011)
0926 03:32:00.471702 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:00.471713 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:00.471744 (+ 31us) server_negotiation.cc:407] Connection header received
0926 03:32:00.471840 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:00.471846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:00.472035 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:00.472727 (+ 692us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:32:00.473539 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.475112 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.478810 (+ 3698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.481013 (+ 2203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.481132 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:00.481334 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:00.482355 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0926 03:32:00.482589 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":307,"threads_started":1}
I20260926 03:32:00.484020 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:00.470827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34011 (local address 127.0.0.1:41514)
0926 03:32:00.471450 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:00.471473 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:00.471621 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:00.472944 (+ 1323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:00.472948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:00.472977 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:32:00.473343 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:00.473360 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.475303 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.475311 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:00.478458 (+ 3147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:00.478477 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.481482 (+ 3005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.481489 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:00.481695 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:00.481735 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 03:32:00.481826 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 03:32:00.481964 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":327,"threads_started":1}
I20260926 03:32:00.488305 7579 tablet_service.cc:1467] Processing CreateTablet for tablet a48e0e93920a4dbdaebc9205cd372231 (DEFAULT_TABLE table=test-table [id=ee4f0f7f97b244cc8740b3bc2e732aad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:32:00.489775 7579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a48e0e93920a4dbdaebc9205cd372231. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:32:00.495929 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:00.473067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53752 (local address 127.0.97.3:36319)
0926 03:32:00.485379 (+ 12312us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:00.485391 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:00.485432 (+ 41us) server_negotiation.cc:407] Connection header received
0926 03:32:00.486070 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:00.486082 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:00.486359 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:00.486810 (+ 451us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:32:00.487330 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.489617 (+ 2287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.493415 (+ 3798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.494825 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.494950 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:00.495139 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:00.495458 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 03:32:00.495622 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12093,"thread_start_us":11886,"threads_started":1}
I20260926 03:32:00.495971 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:00.472205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36319 (local address 127.0.0.1:53752)
0926 03:32:00.484056 (+ 11851us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:00.484436 (+ 380us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:00.484615 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:00.486622 (+ 2007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:00.486630 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:00.486696 (+ 66us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:32:00.487115 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:00.487130 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.489752 (+ 2622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.489759 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:00.493245 (+ 3486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:00.493261 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.495051 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.495057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:00.495242 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:00.495284 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 03:32:00.495388 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 03:32:00.495525 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11606,"mutex_wait_us":16,"spinlock_wait_cycles":2304,"thread_start_us":17564,"threads_started":1}
I20260926 03:32:00.499555 7720 tablet_service.cc:1467] Processing CreateTablet for tablet a48e0e93920a4dbdaebc9205cd372231 (DEFAULT_TABLE table=test-table [id=ee4f0f7f97b244cc8740b3bc2e732aad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:32:00.500900 7720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a48e0e93920a4dbdaebc9205cd372231. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:32:00.504612 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:00.474133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52464 (local address 127.0.97.1:41735)
0926 03:32:00.481423 (+ 7290us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:00.481435 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:00.485613 (+ 4178us) server_negotiation.cc:407] Connection header received
0926 03:32:00.485905 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:00.485917 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:00.486178 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:00.486341 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:32:00.487597 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.489050 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.501465 (+ 12415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.502932 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.503037 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:00.503253 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:00.503923 (+ 670us) server_negotiation.cc:299] Negotiation successful
0926 03:32:00.504165 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7076,"thread_start_us":8090,"threads_started":1}
I20260926 03:32:00.505930 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:00.472696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41735 (local address 127.0.0.1:52464)
0926 03:32:00.485512 (+ 12816us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:00.485530 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:00.485708 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:00.486635 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:00.486639 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:00.486700 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:32:00.487430 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:00.487440 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.489231 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.489235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:00.501260 (+ 12025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:00.501278 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.503409 (+ 2131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.503415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:00.503635 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:00.503691 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 03:32:00.505181 (+ 1490us) client_negotiation.cc:240] Negotiation successful
0926 03:32:00.505398 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12716,"mutex_wait_us":27,"spinlock_wait_cycles":28928,"thread_start_us":11574,"threads_started":1}
I20260926 03:32:00.506991 7822 tablet_bootstrap.cc:492] T a48e0e93920a4dbdaebc9205cd372231 P b61e6b996c96424aa56ab30a31376891: Bootstrap starting.
I20260926 03:32:00.509758 7434 tablet_service.cc:1467] Processing CreateTablet for tablet a48e0e93920a4dbdaebc9205cd372231 (DEFAULT_TABLE table=test-table [id=ee4f0f7f97b244cc8740b3bc2e732aad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:32:00.510697 7822 tablet_bootstrap.cc:654] T a48e0e93920a4dbdaebc9205cd372231 P b61e6b996c96424aa56ab30a31376891: Neither blocks nor log segments found. Creating new log.
I20260926 03:32:00.511365 7434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a48e0e93920a4dbdaebc9205cd372231. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:32:00.512176 7822 log.cc:824] T a48e0e93920a4dbdaebc9205cd372231 P b61e6b996c96424aa56ab30a31376891: Log is configured to *not* fsync() on all Append() calls
I20260926 03:32:00.520650 7822 tablet_bootstrap.cc:492] T a48e0e93920a4dbdaebc9205cd372231 P b61e6b996c96424aa56ab30a31376891: No bootstrap required, opened a new log
I20260926 03:32:00.520830 7824 tablet_bootstrap.cc:492] T a48e0e93920a4dbdaebc9205cd372231 P 91c53a144b0242e491f9e8b8aaef3f02: Bootstrap starting.
I20260926 03:32:00.520933 7822 ts_tablet_manager.cc:1397] T a48e0e93920a4dbdaebc9205cd372231 P b61e6b996c96424aa56ab30a31376891: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20260926 03:32:00.523507 7824 tablet_bootstrap.cc:654] T a48e0e93920a4dbdaebc9205cd372231 P 91c53a144b0242e491f9e8b8aaef3f02: Neither blocks nor log segments found. Creating new log.
I20260926 03:32:00.525363 7824 log.cc:824] T a48e0e93920a4dbdaebc9205cd372231 P 91c53a144b0242e491f9e8b8aaef3f02: Log is configured to *not* fsync() on all Append() calls
I20260926 03:32:00.528205 7824 tablet_bootstrap.cc:492] T a48e0e93920a4dbdaebc9205cd372231 P 91c53a144b0242e491f9e8b8aaef3f02: No bootstrap required, opened a new log
I20260926 03:32:00.528504 7824 ts_tablet_manager.cc:1397] T a48e0e93920a4dbdaebc9205cd372231 P 91c53a144b0242e491f9e8b8aaef3f02: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260926 03:32:00.529292 7826 tablet_bootstrap.cc:492] T a48e0e93920a4dbdaebc9205cd372231 P 6b0027955ae14788b8afa531cbfd18e1: Bootstrap starting.
I20260926 03:32:00.531339 7822 raft_consensus.cc:348] T a48e0e93920a4dbdaebc9205cd372231 P b61e6b996c96424aa56ab30a31376891 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b61e6b996c96424aa56ab30a31376891" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34011 } } peers { permanent_uuid: "91c53a144b0242e491f9e8b8aaef3f02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36319 } } peers { permanent_uuid: "6b0027955ae14788b8afa531cbfd18e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41735 } }
I20260926 03:32:00.531703 7822 raft_consensus.cc:374] T a48e0e93920a4dbdaebc9205cd372231 P b61e6b996c96424aa56ab30a31376891 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:32:00.531800 7822 raft_consensus.cc:729] T a48e0e93920a4dbdaebc9205cd372231 P b61e6b996c96424aa56ab30a31376891 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b61e6b996c96424aa56ab30a31376891, State: Initialized, Role: FOLLOWER
I20260926 03:32:00.532509 7822 consensus_queue.cc:260] T a48e0e93920a4dbdaebc9205cd372231 P b61e6b996c96424aa56ab30a31376891 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b61e6b996c96424aa56ab30a31376891" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34011 } } peers { permanent_uuid: "91c53a144b0242e491f9e8b8aaef3f02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36319 } } peers { permanent_uuid: "6b0027955ae14788b8afa531cbfd18e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41735 } }
I20260926 03:32:00.533195 7826 tablet_bootstrap.cc:654] T a48e0e93920a4dbdaebc9205cd372231 P 6b0027955ae14788b8afa531cbfd18e1: Neither blocks nor log segments found. Creating new log.
I20260926 03:32:00.533631 7644 heartbeater.cc:503] Master 127.0.97.62:46591 was elected leader, sending a full tablet report...
I20260926 03:32:00.534236 7822 ts_tablet_manager.cc:1428] T a48e0e93920a4dbdaebc9205cd372231 P b61e6b996c96424aa56ab30a31376891: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260926 03:32:00.536119 7826 log.cc:824] T a48e0e93920a4dbdaebc9205cd372231 P 6b0027955ae14788b8afa531cbfd18e1: Log is configured to *not* fsync() on all Append() calls
I20260926 03:32:00.540445 7824 raft_consensus.cc:348] T a48e0e93920a4dbdaebc9205cd372231 P 91c53a144b0242e491f9e8b8aaef3f02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b61e6b996c96424aa56ab30a31376891" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34011 } } peers { permanent_uuid: "91c53a144b0242e491f9e8b8aaef3f02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36319 } } peers { permanent_uuid: "6b0027955ae14788b8afa531cbfd18e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41735 } }
I20260926 03:32:00.540904 7824 raft_consensus.cc:374] T a48e0e93920a4dbdaebc9205cd372231 P 91c53a144b0242e491f9e8b8aaef3f02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:32:00.541023 7824 raft_consensus.cc:729] T a48e0e93920a4dbdaebc9205cd372231 P 91c53a144b0242e491f9e8b8aaef3f02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91c53a144b0242e491f9e8b8aaef3f02, State: Initialized, Role: FOLLOWER
I20260926 03:32:00.541677 7826 tablet_bootstrap.cc:492] T a48e0e93920a4dbdaebc9205cd372231 P 6b0027955ae14788b8afa531cbfd18e1: No bootstrap required, opened a new log
I20260926 03:32:00.541651 7824 consensus_queue.cc:260] T a48e0e93920a4dbdaebc9205cd372231 P 91c53a144b0242e491f9e8b8aaef3f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b61e6b996c96424aa56ab30a31376891" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34011 } } peers { permanent_uuid: "91c53a144b0242e491f9e8b8aaef3f02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36319 } } peers { permanent_uuid: "6b0027955ae14788b8afa531cbfd18e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41735 } }
I20260926 03:32:00.542071 7826 ts_tablet_manager.cc:1397] T a48e0e93920a4dbdaebc9205cd372231 P 6b0027955ae14788b8afa531cbfd18e1: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20260926 03:32:00.542893 7785 heartbeater.cc:503] Master 127.0.97.62:46591 was elected leader, sending a full tablet report...
I20260926 03:32:00.544008 7824 ts_tablet_manager.cc:1428] T a48e0e93920a4dbdaebc9205cd372231 P 91c53a144b0242e491f9e8b8aaef3f02: Time spent starting tablet: real 0.015s user 0.011s sys 0.002s
I20260926 03:32:00.551322 7826 raft_consensus.cc:348] T a48e0e93920a4dbdaebc9205cd372231 P 6b0027955ae14788b8afa531cbfd18e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b61e6b996c96424aa56ab30a31376891" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34011 } } peers { permanent_uuid: "91c53a144b0242e491f9e8b8aaef3f02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36319 } } peers { permanent_uuid: "6b0027955ae14788b8afa531cbfd18e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41735 } }
I20260926 03:32:00.551676 7826 raft_consensus.cc:374] T a48e0e93920a4dbdaebc9205cd372231 P 6b0027955ae14788b8afa531cbfd18e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:32:00.551797 7826 raft_consensus.cc:729] T a48e0e93920a4dbdaebc9205cd372231 P 6b0027955ae14788b8afa531cbfd18e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b0027955ae14788b8afa531cbfd18e1, State: Initialized, Role: FOLLOWER
I20260926 03:32:00.552337 7826 consensus_queue.cc:260] T a48e0e93920a4dbdaebc9205cd372231 P 6b0027955ae14788b8afa531cbfd18e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b61e6b996c96424aa56ab30a31376891" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34011 } } peers { permanent_uuid: "91c53a144b0242e491f9e8b8aaef3f02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36319 } } peers { permanent_uuid: "6b0027955ae14788b8afa531cbfd18e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41735 } }
I20260926 03:32:00.553960 7826 ts_tablet_manager.cc:1428] T a48e0e93920a4dbdaebc9205cd372231 P 6b0027955ae14788b8afa531cbfd18e1: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
W20260926 03:32:00.622630 7789 tablet.cc:2369] T a48e0e93920a4dbdaebc9205cd372231 P 91c53a144b0242e491f9e8b8aaef3f02: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:32:00.630952 7502 tablet.cc:2369] T a48e0e93920a4dbdaebc9205cd372231 P 6b0027955ae14788b8afa531cbfd18e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:32:00.782747 7647 tablet.cc:2369] T a48e0e93920a4dbdaebc9205cd372231 P b61e6b996c96424aa56ab30a31376891: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:32:00.795322 7827 raft_consensus.cc:482] T a48e0e93920a4dbdaebc9205cd372231 P b61e6b996c96424aa56ab30a31376891 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:32:00.795603 7827 raft_consensus.cc:504] T a48e0e93920a4dbdaebc9205cd372231 P b61e6b996c96424aa56ab30a31376891 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b61e6b996c96424aa56ab30a31376891" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34011 } } peers { permanent_uuid: "91c53a144b0242e491f9e8b8aaef3f02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36319 } } peers { permanent_uuid: "6b0027955ae14788b8afa531cbfd18e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41735 } }
I20260926 03:32:00.797465 7827 leader_election.cc:288] T a48e0e93920a4dbdaebc9205cd372231 P b61e6b996c96424aa56ab30a31376891 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91c53a144b0242e491f9e8b8aaef3f02 (127.0.97.3:36319), 6b0027955ae14788b8afa531cbfd18e1 (127.0.97.1:41735)
I20260926 03:32:00.804348 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:00.797895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47035 (local address 127.0.97.3:36319)
0926 03:32:00.798133 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:00.798141 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:00.798471 (+ 330us) server_negotiation.cc:407] Connection header received
0926 03:32:00.798683 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:00.798691 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:00.798772 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:00.798853 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:32:00.799854 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.800673 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.802515 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.803413 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.803509 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:00.803581 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:00.803972 (+ 391us) server_negotiation.cc:299] Negotiation successful
0926 03:32:00.804110 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 03:32:00.804652 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:00.797666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36319 (local address 127.0.97.2:47035)
0926 03:32:00.798345 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:00.798365 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:00.798541 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:00.798890 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:00.798894 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:00.798919 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:32:00.799708 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:00.799721 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.800823 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.800828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:00.802364 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:00.802378 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.803630 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.803636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:00.803796 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:00.803843 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 03:32:00.803931 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 03:32:00.804063 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":1,"thread_start_us":340,"threads_started":1}
I20260926 03:32:00.804750 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:00.797952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40363 (local address 127.0.97.1:41735)
0926 03:32:00.798129 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:00.798136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:00.798747 (+ 611us) server_negotiation.cc:407] Connection header received
0926 03:32:00.798905 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:00.798910 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:00.799020 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:00.799129 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:32:00.800055 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.801079 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.802877 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.803767 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.803911 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:00.804021 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:00.804357 (+ 336us) server_negotiation.cc:299] Negotiation successful
0926 03:32:00.804509 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 03:32:00.805124 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:00.797746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41735 (local address 127.0.97.2:40363)
0926 03:32:00.798634 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:00.798650 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:00.798767 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:00.799153 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:00.799156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:00.799173 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:32:00.799885 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:00.799897 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.801203 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.801207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:00.802724 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:00.802737 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.804011 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.804018 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:00.804182 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:00.804215 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 03:32:00.804549 (+ 334us) client_negotiation.cc:240] Negotiation successful
0926 03:32:00.804689 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"mutex_wait_us":49,"thread_start_us":545,"threads_started":1}
I20260926 03:32:00.805742 7740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48e0e93920a4dbdaebc9205cd372231" candidate_uuid: "b61e6b996c96424aa56ab30a31376891" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91c53a144b0242e491f9e8b8aaef3f02" is_pre_election: true
I20260926 03:32:00.806030 7454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48e0e93920a4dbdaebc9205cd372231" candidate_uuid: "b61e6b996c96424aa56ab30a31376891" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b0027955ae14788b8afa531cbfd18e1" is_pre_election: true
I20260926 03:32:00.806258 7740 raft_consensus.cc:2454] T a48e0e93920a4dbdaebc9205cd372231 P 91c53a144b0242e491f9e8b8aaef3f02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b61e6b996c96424aa56ab30a31376891 in term 0.
I20260926 03:32:00.806501 7454 raft_consensus.cc:2454] T a48e0e93920a4dbdaebc9205cd372231 P 6b0027955ae14788b8afa531cbfd18e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b61e6b996c96424aa56ab30a31376891 in term 0.
I20260926 03:32:00.806903 7535 leader_election.cc:302] T a48e0e93920a4dbdaebc9205cd372231 P b61e6b996c96424aa56ab30a31376891 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91c53a144b0242e491f9e8b8aaef3f02, b61e6b996c96424aa56ab30a31376891; no voters:
I20260926 03:32:00.807287 7827 raft_consensus.cc:2781] T a48e0e93920a4dbdaebc9205cd372231 P b61e6b996c96424aa56ab30a31376891 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:32:00.807428 7827 raft_consensus.cc:482] T a48e0e93920a4dbdaebc9205cd372231 P b61e6b996c96424aa56ab30a31376891 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:32:00.807521 7827 raft_consensus.cc:3037] T a48e0e93920a4dbdaebc9205cd372231 P b61e6b996c96424aa56ab30a31376891 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:32:00.809674 7827 raft_consensus.cc:504] T a48e0e93920a4dbdaebc9205cd372231 P b61e6b996c96424aa56ab30a31376891 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b61e6b996c96424aa56ab30a31376891" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34011 } } peers { permanent_uuid: "91c53a144b0242e491f9e8b8aaef3f02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36319 } } peers { permanent_uuid: "6b0027955ae14788b8afa531cbfd18e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41735 } }
I20260926 03:32:00.810243 7827 leader_election.cc:288] T a48e0e93920a4dbdaebc9205cd372231 P b61e6b996c96424aa56ab30a31376891 [CANDIDATE]: Term 1 election: Requested vote from peers 91c53a144b0242e491f9e8b8aaef3f02 (127.0.97.3:36319), 6b0027955ae14788b8afa531cbfd18e1 (127.0.97.1:41735)
I20260926 03:32:00.810621 7740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48e0e93920a4dbdaebc9205cd372231" candidate_uuid: "b61e6b996c96424aa56ab30a31376891" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91c53a144b0242e491f9e8b8aaef3f02"
I20260926 03:32:00.810734 7454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48e0e93920a4dbdaebc9205cd372231" candidate_uuid: "b61e6b996c96424aa56ab30a31376891" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b0027955ae14788b8afa531cbfd18e1"
I20260926 03:32:00.810886 7740 raft_consensus.cc:3037] T a48e0e93920a4dbdaebc9205cd372231 P 91c53a144b0242e491f9e8b8aaef3f02 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:32:00.810994 7454 raft_consensus.cc:3037] T a48e0e93920a4dbdaebc9205cd372231 P 6b0027955ae14788b8afa531cbfd18e1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:32:00.813477 7740 raft_consensus.cc:2454] T a48e0e93920a4dbdaebc9205cd372231 P 91c53a144b0242e491f9e8b8aaef3f02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b61e6b996c96424aa56ab30a31376891 in term 1.
I20260926 03:32:00.813794 7454 raft_consensus.cc:2454] T a48e0e93920a4dbdaebc9205cd372231 P 6b0027955ae14788b8afa531cbfd18e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b61e6b996c96424aa56ab30a31376891 in term 1.
I20260926 03:32:00.813902 7535 leader_election.cc:302] T a48e0e93920a4dbdaebc9205cd372231 P b61e6b996c96424aa56ab30a31376891 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91c53a144b0242e491f9e8b8aaef3f02, b61e6b996c96424aa56ab30a31376891; no voters:
I20260926 03:32:00.814203 7827 raft_consensus.cc:2781] T a48e0e93920a4dbdaebc9205cd372231 P b61e6b996c96424aa56ab30a31376891 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:32:00.814634 7827 raft_consensus.cc:686] T a48e0e93920a4dbdaebc9205cd372231 P b61e6b996c96424aa56ab30a31376891 [term 1 LEADER]: Becoming Leader. State: Replica: b61e6b996c96424aa56ab30a31376891, State: Running, Role: LEADER
I20260926 03:32:00.815201 7827 consensus_queue.cc:237] T a48e0e93920a4dbdaebc9205cd372231 P b61e6b996c96424aa56ab30a31376891 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b61e6b996c96424aa56ab30a31376891" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34011 } } peers { permanent_uuid: "91c53a144b0242e491f9e8b8aaef3f02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36319 } } peers { permanent_uuid: "6b0027955ae14788b8afa531cbfd18e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41735 } }
I20260926 03:32:00.820218 6534 catalog_manager.cc:5495] T a48e0e93920a4dbdaebc9205cd372231 P b61e6b996c96424aa56ab30a31376891 reported cstate change: term changed from 0 to 1, leader changed from <none> to b61e6b996c96424aa56ab30a31376891 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "b61e6b996c96424aa56ab30a31376891" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b61e6b996c96424aa56ab30a31376891" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91c53a144b0242e491f9e8b8aaef3f02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b0027955ae14788b8afa531cbfd18e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260926 03:32:00.861660 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:00.855923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41524 (local address 127.0.97.2:34011)
0926 03:32:00.856122 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:00.856128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:00.856235 (+ 107us) server_negotiation.cc:407] Connection header received
0926 03:32:00.856461 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:00.856465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:00.856545 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:00.856763 (+ 218us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:32:00.857113 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.858248 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.859525 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.859911 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.860045 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:00.860503 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:32:00.861048 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:32:00.861199 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:00.861293 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 03:32:00.861390 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 03:32:00.869439 7740 raft_consensus.cc:1261] T a48e0e93920a4dbdaebc9205cd372231 P 91c53a144b0242e491f9e8b8aaef3f02 [term 1 FOLLOWER]: Refusing update from remote peer b61e6b996c96424aa56ab30a31376891: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:32:00.869453 7454 raft_consensus.cc:1261] T a48e0e93920a4dbdaebc9205cd372231 P 6b0027955ae14788b8afa531cbfd18e1 [term 1 FOLLOWER]: Refusing update from remote peer b61e6b996c96424aa56ab30a31376891: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:32:00.870360 7833 consensus_queue.cc:1035] T a48e0e93920a4dbdaebc9205cd372231 P b61e6b996c96424aa56ab30a31376891 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c53a144b0242e491f9e8b8aaef3f02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:32:00.870675 7827 consensus_queue.cc:1035] T a48e0e93920a4dbdaebc9205cd372231 P b61e6b996c96424aa56ab30a31376891 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b0027955ae14788b8afa531cbfd18e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:32:00.886310 7836 mvcc.cc:205] Tried to move back new op lower bound from 7333451861471444992 to 7333451861262696448. Current Snapshot: MvccSnapshot[applied={T|T < 7333451861471444992}]
I20260926 03:32:00.890697 7837 mvcc.cc:205] Tried to move back new op lower bound from 7333451861471444992 to 7333451861262696448. Current Snapshot: MvccSnapshot[applied={T|T < 7333451861471444992}]
I20260926 03:32:00.902601 6535 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:32:00.902843 6535 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:32:00.908361 6535 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aaf83112ebc946f5b71d630f6d3454b0: Sending DeleteTablet for 3 replicas of tablet a48e0e93920a4dbdaebc9205cd372231
I20260926 03:32:00.909430 7579 tablet_service.cc:1514] Processing DeleteTablet for tablet a48e0e93920a4dbdaebc9205cd372231 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:32:00 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41514
I20260926 03:32:00.909914 7720 tablet_service.cc:1514] Processing DeleteTablet for tablet a48e0e93920a4dbdaebc9205cd372231 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:32:00 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53752
I20260926 03:32:00.910857 7845 tablet_replica.cc:330] T a48e0e93920a4dbdaebc9205cd372231 P b61e6b996c96424aa56ab30a31376891: stopping tablet replica
I20260926 03:32:00.911247 7845 raft_consensus.cc:2229] T a48e0e93920a4dbdaebc9205cd372231 P b61e6b996c96424aa56ab30a31376891 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:32:00.911844 7434 tablet_service.cc:1514] Processing DeleteTablet for tablet a48e0e93920a4dbdaebc9205cd372231 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:32:00 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52464
I20260926 03:32:00.911957 7845 raft_consensus.cc:2258] T a48e0e93920a4dbdaebc9205cd372231 P b61e6b996c96424aa56ab30a31376891 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:32:00.914525 7845 ts_tablet_manager.cc:1905] T a48e0e93920a4dbdaebc9205cd372231 P b61e6b996c96424aa56ab30a31376891: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:32:00.920047 7847 tablet_replica.cc:330] T a48e0e93920a4dbdaebc9205cd372231 P 6b0027955ae14788b8afa531cbfd18e1: stopping tablet replica
I20260926 03:32:00.920480 7847 raft_consensus.cc:2229] T a48e0e93920a4dbdaebc9205cd372231 P 6b0027955ae14788b8afa531cbfd18e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:32:00.921021 7847 raft_consensus.cc:2258] T a48e0e93920a4dbdaebc9205cd372231 P 6b0027955ae14788b8afa531cbfd18e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:32:00.923548 7847 ts_tablet_manager.cc:1905] T a48e0e93920a4dbdaebc9205cd372231 P 6b0027955ae14788b8afa531cbfd18e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:32:00.923628 7846 tablet_replica.cc:330] T a48e0e93920a4dbdaebc9205cd372231 P 91c53a144b0242e491f9e8b8aaef3f02: stopping tablet replica
I20260926 03:32:00.924876 7845 ts_tablet_manager.cc:1918] T a48e0e93920a4dbdaebc9205cd372231 P b61e6b996c96424aa56ab30a31376891: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:32:00.925124 7846 raft_consensus.cc:2229] T a48e0e93920a4dbdaebc9205cd372231 P 91c53a144b0242e491f9e8b8aaef3f02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:32:00.925499 7845 log.cc:1196] T a48e0e93920a4dbdaebc9205cd372231 P b61e6b996c96424aa56ab30a31376891: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-1/wal/wals/a48e0e93920a4dbdaebc9205cd372231
I20260926 03:32:00.925601 7846 raft_consensus.cc:2258] T a48e0e93920a4dbdaebc9205cd372231 P 91c53a144b0242e491f9e8b8aaef3f02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:32:00.926550 7845 ts_tablet_manager.cc:1939] T a48e0e93920a4dbdaebc9205cd372231 P b61e6b996c96424aa56ab30a31376891: Deleting consensus metadata
I20260926 03:32:00.928339 6520 catalog_manager.cc:4841] TS b61e6b996c96424aa56ab30a31376891 (127.0.97.2:34011): tablet a48e0e93920a4dbdaebc9205cd372231 (table test-table [id=ee4f0f7f97b244cc8740b3bc2e732aad]) successfully deleted
I20260926 03:32:00.929862 7846 ts_tablet_manager.cc:1905] T a48e0e93920a4dbdaebc9205cd372231 P 91c53a144b0242e491f9e8b8aaef3f02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:32:00.931195 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 6503
I20260926 03:32:00.932916 6565 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:32:00.937211 7846 ts_tablet_manager.cc:1918] T a48e0e93920a4dbdaebc9205cd372231 P 91c53a144b0242e491f9e8b8aaef3f02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:32:00.937374 7846 log.cc:1196] T a48e0e93920a4dbdaebc9205cd372231 P 91c53a144b0242e491f9e8b8aaef3f02: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-2/wal/wals/a48e0e93920a4dbdaebc9205cd372231
I20260926 03:32:00.937915 7846 ts_tablet_manager.cc:1939] T a48e0e93920a4dbdaebc9205cd372231 P 91c53a144b0242e491f9e8b8aaef3f02: Deleting consensus metadata
I20260926 03:32:00.940075 7847 ts_tablet_manager.cc:1918] T a48e0e93920a4dbdaebc9205cd372231 P 6b0027955ae14788b8afa531cbfd18e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:32:00.940253 7847 log.cc:1196] T a48e0e93920a4dbdaebc9205cd372231 P 6b0027955ae14788b8afa531cbfd18e1: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-0/wal/wals/a48e0e93920a4dbdaebc9205cd372231
I20260926 03:32:00.940795 7847 ts_tablet_manager.cc:1939] T a48e0e93920a4dbdaebc9205cd372231 P 6b0027955ae14788b8afa531cbfd18e1: Deleting consensus metadata
I20260926 03:32:00.947398 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:00.913601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44228 (local address 127.0.97.61:42461)
0926 03:32:00.913794 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:00.913800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:00.924485 (+ 10685us) server_negotiation.cc:407] Connection header received
0926 03:32:00.925161 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:00.925168 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:00.925254 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:00.925365 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:00.929259 (+ 3894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.930318 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.933555 (+ 3237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.934476 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.934601 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:00.939432 (+ 4831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:00.939439 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:00.939445 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:00.939490 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:00.943917 (+ 4427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:00.945149 (+ 1232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:00.945157 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:00.945160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:00.945309 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:00.945763 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:00.945770 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:00.945776 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:00.946211 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:00.946453 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:00.946836 (+ 383us) server_negotiation.cc:299] Negotiation successful
0926 03:32:00.947055 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 03:32:00.963999 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:00.930533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36180 (local address 127.0.97.60:36639)
0926 03:32:00.930712 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:00.930718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:00.947525 (+ 16807us) server_negotiation.cc:407] Connection header received
0926 03:32:00.947787 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:00.947794 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:00.947886 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:00.947958 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:00.949803 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.950973 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.952749 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:00.953374 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:00.953512 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:00.957244 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:00.957252 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:00.957256 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:00.957293 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:00.961033 (+ 3740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:00.961825 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:00.961831 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:00.961834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:00.961964 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:00.962384 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:00.962390 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:00.962395 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:00.962722 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:00.962971 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:00.963291 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 03:32:00.963564 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 03:32:01.077608 6519 catalog_manager.cc:4841] TS 6b0027955ae14788b8afa531cbfd18e1 (127.0.97.1:41735): tablet a48e0e93920a4dbdaebc9205cd372231 (table test-table [id=ee4f0f7f97b244cc8740b3bc2e732aad]) successfully deleted
W20260926 03:32:01.077875 6519 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.97.1:41735, user_credentials={real_user=slave}} blocked reactor thread for 135481us
I20260926 03:32:01.078394 6522 catalog_manager.cc:4841] TS 91c53a144b0242e491f9e8b8aaef3f02 (127.0.97.3:36319): tablet a48e0e93920a4dbdaebc9205cd372231 (table test-table [id=ee4f0f7f97b244cc8740b3bc2e732aad]) successfully deleted
W20260926 03:32:01.078558 6522 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.97.3:36319, user_credentials={real_user=slave}} blocked reactor thread for 138886us
I20260926 03:32:01.078677 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 6503
I20260926 03:32:01.129801 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:01.113668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36192 (local address 127.0.97.60:36639)
0926 03:32:01.113812 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:01.113818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:01.113922 (+ 104us) server_negotiation.cc:407] Connection header received
0926 03:32:01.114114 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:01.114119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:01.114208 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:01.114323 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:01.115897 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:01.116865 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:01.118867 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:01.119516 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:01.119619 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:01.123191 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:01.123196 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:01.123199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:01.123230 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:01.126637 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:01.127524 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:01.127532 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:01.127536 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:01.127662 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:01.128071 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:01.128076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:01.128080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:01.128402 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:01.128645 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:01.129230 (+ 585us) server_negotiation.cc:299] Negotiation successful
0926 03:32:01.129444 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 03:32:01.130865 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:01.112279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44240 (local address 127.0.97.61:42461)
0926 03:32:01.112505 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:01.112515 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:01.113525 (+ 1010us) server_negotiation.cc:407] Connection header received
0926 03:32:01.113822 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:01.113830 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:01.113910 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:01.113982 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:01.116418 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:01.117350 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:01.118843 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:01.119287 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:01.119432 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:01.123151 (+ 3719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:01.123158 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:01.123176 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:01.123215 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:01.127230 (+ 4015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:01.128154 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:01.128161 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:01.128165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:01.128318 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:01.128824 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:01.128830 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:01.128834 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:01.129191 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:01.129404 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:01.130300 (+ 896us) server_negotiation.cc:299] Negotiation successful
0926 03:32:01.130540 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 03:32:01.893915 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:01.893010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46591 (local address 127.0.97.1:48783)
0926 03:32:01.893523 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:01.893686 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":251,"threads_started":1}
W20260926 03:32:01.894413 7499 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46591 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:46591: connect: Connection refused (error 111)
I20260926 03:32:01.894654 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:01.893497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46591 (local address 127.0.97.2:53091)
0926 03:32:01.894282 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:01.894409 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":314,"threads_started":1}
W20260926 03:32:01.894994 7644 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46591 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:46591: connect: Connection refused (error 111)
I20260926 03:32:01.899499 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:01.898381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46591 (local address 127.0.97.3:37343)
0926 03:32:01.899228 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:01.899352 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":648,"threads_started":1}
W20260926 03:32:01.899907 7785 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46591 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:46591: connect: Connection refused (error 111)
I20260926 03:32:02.409297 7859 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aaf83112ebc946f5b71d630f6d3454b0)
I20260926 03:32:02.409646 7859 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: false } } peers { permanent_uuid: "5a56fb218c4e482fac40da5c0be63a91" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36639 } attrs { promote: false } }
I20260926 03:32:02.409816 7860 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aaf83112ebc946f5b71d630f6d3454b0)
I20260926 03:32:02.410079 7860 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: false } } peers { permanent_uuid: "5a56fb218c4e482fac40da5c0be63a91" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36639 } attrs { promote: false } }
I20260926 03:32:02.411057 7859 leader_election.cc:288] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aaf83112ebc946f5b71d630f6d3454b0 (127.0.97.62:46591), 5a56fb218c4e482fac40da5c0be63a91 (127.0.97.60:36639)
I20260926 03:32:02.411471 7860 leader_election.cc:288] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aaf83112ebc946f5b71d630f6d3454b0 (127.0.97.62:46591), 8692ab0d52964a07803500c4080166c4 (127.0.97.61:42461)
I20260926 03:32:02.412663 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.411295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46591 (local address 127.0.0.1:43064)
0926 03:32:02.412118 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:02.412340 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":528,"threads_started":1}
W20260926 03:32:02.413666 7119 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46591: connect: Connection refused (error 111)
I20260926 03:32:02.414845 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.411553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46591 (local address 127.0.0.1:43078)
0926 03:32:02.414322 (+ 2769us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:02.414537 (+ 215us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2620,"threads_started":1}
W20260926 03:32:02.415701 7306 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46591: connect: Connection refused (error 111)
I20260926 03:32:02.426821 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.411876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44716 (local address 127.0.97.61:42461)
0926 03:32:02.413868 (+ 1992us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:02.413874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:02.415183 (+ 1309us) server_negotiation.cc:407] Connection header received
0926 03:32:02.415844 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:02.415853 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:02.415947 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:02.416030 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:32:02.417039 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:02.418693 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:02.422963 (+ 4270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:02.424876 (+ 1913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:02.425055 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:02.425206 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:02.426033 (+ 827us) server_negotiation.cc:299] Negotiation successful
0926 03:32:02.426245 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1886,"thread_start_us":1706,"threads_started":1}
I20260926 03:32:02.427927 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.411618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42461 (local address 127.0.0.1:44716)
0926 03:32:02.415067 (+ 3449us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:02.415088 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:02.415642 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:02.416357 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:02.416365 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:02.416452 (+ 87us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:32:02.416809 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:02.416825 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:02.419871 (+ 3046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:02.419876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:02.422696 (+ 2820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:02.422724 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:02.425463 (+ 2739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:02.425473 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:02.425781 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:02.425836 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 03:32:02.427099 (+ 1263us) client_negotiation.cc:240] Negotiation successful
0926 03:32:02.427336 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3333,"thread_start_us":2412,"threads_started":1}
I20260926 03:32:02.429100 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a56fb218c4e482fac40da5c0be63a91" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8692ab0d52964a07803500c4080166c4" is_pre_election: true
I20260926 03:32:02.429570 7143 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a56fb218c4e482fac40da5c0be63a91 in term 1.
I20260926 03:32:02.429636 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.427745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46591 (local address 127.0.0.1:43104)
0926 03:32:02.428349 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:02.428759 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
W20260926 03:32:02.429966 7306 leader_election.cc:334] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aaf83112ebc946f5b71d630f6d3454b0 (127.0.97.62:46591): Network error: Client connection negotiation failed: client connection to 127.0.97.62:46591: connect: Connection refused (error 111)
I20260926 03:32:02.430274 7304 leader_election.cc:302] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a56fb218c4e482fac40da5c0be63a91, 8692ab0d52964a07803500c4080166c4; no voters: aaf83112ebc946f5b71d630f6d3454b0
I20260926 03:32:02.430716 7860 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 03:32:02.430832 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.430261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46591 (local address 127.0.0.1:43108)
0926 03:32:02.430473 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:02.430633 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 03:32:02.430845 7860 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [term 1 FOLLOWER]: Starting leader election (detected failure of leader aaf83112ebc946f5b71d630f6d3454b0)
I20260926 03:32:02.430946 7860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [term 1 FOLLOWER]: Advancing to term 2
W20260926 03:32:02.431021 7119 leader_election.cc:334] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aaf83112ebc946f5b71d630f6d3454b0 (127.0.97.62:46591): Network error: Client connection negotiation failed: client connection to 127.0.97.62:46591: connect: Connection refused (error 111)
I20260926 03:32:02.434054 7860 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: false } } peers { permanent_uuid: "5a56fb218c4e482fac40da5c0be63a91" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36639 } attrs { promote: false } }
I20260926 03:32:02.434726 7860 leader_election.cc:288] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [CANDIDATE]: Term 2 election: Requested vote from peers aaf83112ebc946f5b71d630f6d3454b0 (127.0.97.62:46591), 8692ab0d52964a07803500c4080166c4 (127.0.97.61:42461)
I20260926 03:32:02.435284 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.434850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46591 (local address 127.0.0.1:43110)
0926 03:32:02.435036 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:02.435125 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 03:32:02.435331 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a56fb218c4e482fac40da5c0be63a91" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8692ab0d52964a07803500c4080166c4"
I20260926 03:32:02.435550 7143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [term 1 FOLLOWER]: Advancing to term 2
I20260926 03:32:02.435824 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.412195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36639 (local address 127.0.0.1:35912)
0926 03:32:02.412909 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:02.412932 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:02.413090 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:02.428670 (+ 15580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:02.428675 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:02.428704 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:32:02.429442 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:02.429459 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:02.431575 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:02.431581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:02.433617 (+ 2036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:02.433632 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:02.435002 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:02.435009 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:02.435224 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:02.435260 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 03:32:02.435359 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 03:32:02.435500 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":1149,"threads_started":1}
I20260926 03:32:02.436251 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.414319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35912 (local address 127.0.97.60:36639)
0926 03:32:02.427686 (+ 13367us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:02.427692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:02.427711 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:32:02.427771 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:02.427775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:02.427854 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:02.428768 (+ 914us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:32:02.429998 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:02.431396 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:02.433769 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:02.434773 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:02.435644 (+ 871us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:02.435748 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:02.435799 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 03:32:02.435943 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13217,"thread_start_us":13618,"threads_started":1}
I20260926 03:32:02.438076 7143 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a56fb218c4e482fac40da5c0be63a91 in term 2.
I20260926 03:32:02.438544 7304 leader_election.cc:302] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a56fb218c4e482fac40da5c0be63a91, 8692ab0d52964a07803500c4080166c4; no voters:
I20260926 03:32:02.438771 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.438362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46591 (local address 127.0.0.1:43126)
0926 03:32:02.438514 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:02.438599 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 03:32:02.438673 7331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8692ab0d52964a07803500c4080166c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5a56fb218c4e482fac40da5c0be63a91" is_pre_election: true
W20260926 03:32:02.439003 7306 leader_election.cc:334] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer aaf83112ebc946f5b71d630f6d3454b0 (127.0.97.62:46591): Network error: Client connection negotiation failed: client connection to 127.0.97.62:46591: connect: Connection refused (error 111)
I20260926 03:32:02.439061 7331 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8692ab0d52964a07803500c4080166c4 in current term 2: Already voted for candidate 5a56fb218c4e482fac40da5c0be63a91 in this term.
I20260926 03:32:02.439248 7860 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 03:32:02.439750 7860 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [term 2 LEADER]: Becoming Leader. State: Replica: 5a56fb218c4e482fac40da5c0be63a91, State: Running, Role: LEADER
I20260926 03:32:02.439786 7119 leader_election.cc:302] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8692ab0d52964a07803500c4080166c4; no voters: 5a56fb218c4e482fac40da5c0be63a91, aaf83112ebc946f5b71d630f6d3454b0
I20260926 03:32:02.440148 7859 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 03:32:02.440213 7860 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: false } } peers { permanent_uuid: "5a56fb218c4e482fac40da5c0be63a91" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36639 } attrs { promote: false } }
I20260926 03:32:02.442431 7869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a56fb218c4e482fac40da5c0be63a91. Latest consensus state: current_term: 2 leader_uuid: "5a56fb218c4e482fac40da5c0be63a91" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: false } } peers { permanent_uuid: "5a56fb218c4e482fac40da5c0be63a91" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36639 } attrs { promote: false } } }
I20260926 03:32:02.442883 7869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [sys.catalog]: This master's current role is: LEADER
I20260926 03:32:02.443521 7871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:32:02.447536 7871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:32:02.448339 7871 catalog_manager.cc:1259] Loaded cluster ID: 517c6d59af29429eb0c555bedc2befa8
I20260926 03:32:02.448408 7871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:32:02.449725 7871 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:32:02.450502 7871 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91: Loaded TSK: 0
I20260926 03:32:02.451298 7871 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:32:02.488250 7320 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:32:02.495788 7320 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:32:02.499773 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.499345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46591 (local address 127.0.0.1:43142)
0926 03:32:02.499526 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:02.499600 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 03:32:02.499979 7143 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [term 2 FOLLOWER]: Refusing update from remote peer 5a56fb218c4e482fac40da5c0be63a91: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 03:32:02.500877 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.500457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46591 (local address 127.0.0.1:43152)
0926 03:32:02.500656 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:02.500724 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 03:32:02.500857 7869 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260926 03:32:02.501168 7306 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 -> Peer aaf83112ebc946f5b71d630f6d3454b0 (127.0.97.62:46591): Couldn't send request to peer aaf83112ebc946f5b71d630f6d3454b0. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46591: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 03:32:02.503549 7859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a56fb218c4e482fac40da5c0be63a91. Latest consensus state: current_term: 2 leader_uuid: "5a56fb218c4e482fac40da5c0be63a91" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: false } } peers { permanent_uuid: "5a56fb218c4e482fac40da5c0be63a91" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36639 } attrs { promote: false } } }
I20260926 03:32:02.503862 7859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:32:02.505448 7860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "5a56fb218c4e482fac40da5c0be63a91" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: false } } peers { permanent_uuid: "5a56fb218c4e482fac40da5c0be63a91" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36639 } attrs { promote: false } } }
I20260926 03:32:02.505774 7860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91 [sys.catalog]: This master's current role is: LEADER
I20260926 03:32:02.505738 7873 mvcc.cc:205] Tried to move back new op lower bound from 7333451868153118720 to 7333451867919761408. Current Snapshot: MvccSnapshot[applied={T|T < 7333451868153118720}]
I20260926 03:32:02.506366 7859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "5a56fb218c4e482fac40da5c0be63a91" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "aaf83112ebc946f5b71d630f6d3454b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46591 } } peers { permanent_uuid: "8692ab0d52964a07803500c4080166c4" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42461 } attrs { promote: false } } peers { permanent_uuid: "5a56fb218c4e482fac40da5c0be63a91" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36639 } attrs { promote: false } } }
I20260926 03:32:02.506628 7859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8692ab0d52964a07803500c4080166c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:32:02.520419 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.513647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36319 (local address 127.0.0.1:34314)
0926 03:32:02.513805 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:02.513826 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:02.513946 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:02.514652 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:02.514657 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:02.514685 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:32:02.514912 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:02.514920 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:02.516356 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:02.516362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:02.517934 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:02.517987 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:02.519740 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:02.519746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:02.519948 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:02.519995 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 03:32:02.520064 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 03:32:02.520184 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 03:32:02.521631 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.513949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34314 (local address 127.0.97.3:36319)
0926 03:32:02.514340 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:02.514344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:02.514358 (+ 14us) server_negotiation.cc:407] Connection header received
0926 03:32:02.514409 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:02.514412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:02.514463 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:02.515080 (+ 617us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:32:02.515250 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:02.516210 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:02.518207 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:02.519505 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:02.521044 (+ 1539us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:02.521156 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:02.521210 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 03:32:02.521344 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":211,"threads_started":1}
I20260926 03:32:02.522547 7720 tablet_service.cc:1467] Processing CreateTablet for tablet d8e2cfc16e394c53b2e9e8bfcf6c9902 (DEFAULT_TABLE table=test-table [id=2d4a4801a91d4ebb8c894c379f13b4c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:32:02.523046 7720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8e2cfc16e394c53b2e9e8bfcf6c9902. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:32:02.525260 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.513909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41735 (local address 127.0.0.1:56226)
0926 03:32:02.514221 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:02.514238 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:02.514334 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:02.515830 (+ 1496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:02.515838 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:02.515871 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:32:02.516136 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:02.516150 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:02.518384 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:02.518393 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:02.520979 (+ 2586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:02.520997 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:02.524126 (+ 3129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:02.524135 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:02.524431 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:02.524486 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 03:32:02.524598 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 03:32:02.524805 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200}
I20260926 03:32:02.527697 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.514934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56226 (local address 127.0.97.1:41735)
0926 03:32:02.515436 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:02.515442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:02.515464 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:32:02.515549 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:02.515556 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:02.515646 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:02.516303 (+ 657us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:32:02.516499 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:02.518179 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:02.521517 (+ 3338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:02.523903 (+ 2386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:02.526738 (+ 2835us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:02.527000 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:02.527098 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 03:32:02.527352 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":261,"threads_started":1}
I20260926 03:32:02.528138 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.514565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34011 (local address 127.0.0.1:48080)
0926 03:32:02.520692 (+ 6127us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:02.520710 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:02.520841 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:02.521234 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:02.521238 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:02.521261 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:32:02.521424 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:02.521435 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:02.522795 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:02.522800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:02.524731 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:02.524745 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:02.527387 (+ 2642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:02.527393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:02.527623 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:02.527656 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 03:32:02.527722 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 03:32:02.527854 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6008,"thread_start_us":12550,"threads_started":1}
I20260926 03:32:02.528537 7434 tablet_service.cc:1467] Processing CreateTablet for tablet d8e2cfc16e394c53b2e9e8bfcf6c9902 (DEFAULT_TABLE table=test-table [id=2d4a4801a91d4ebb8c894c379f13b4c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:32:02.528944 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.514781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48080 (local address 127.0.97.2:34011)
0926 03:32:02.515243 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:02.515249 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:02.520799 (+ 5550us) server_negotiation.cc:407] Connection header received
0926 03:32:02.521014 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:02.521019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:02.521093 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:02.521567 (+ 474us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:32:02.521757 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:02.522606 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:02.525870 (+ 3264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:02.527218 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:02.528369 (+ 1151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:02.528464 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:02.528512 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 03:32:02.528683 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":243,"threads_started":1}
I20260926 03:32:02.529002 7434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8e2cfc16e394c53b2e9e8bfcf6c9902. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:32:02.529721 7579 tablet_service.cc:1467] Processing CreateTablet for tablet d8e2cfc16e394c53b2e9e8bfcf6c9902 (DEFAULT_TABLE table=test-table [id=2d4a4801a91d4ebb8c894c379f13b4c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:32:02.530107 7579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8e2cfc16e394c53b2e9e8bfcf6c9902. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:32:02.534266 7881 tablet_bootstrap.cc:492] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 6b0027955ae14788b8afa531cbfd18e1: Bootstrap starting.
I20260926 03:32:02.535562 7882 tablet_bootstrap.cc:492] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P b61e6b996c96424aa56ab30a31376891: Bootstrap starting.
I20260926 03:32:02.536172 7881 tablet_bootstrap.cc:654] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 6b0027955ae14788b8afa531cbfd18e1: Neither blocks nor log segments found. Creating new log.
I20260926 03:32:02.536370 7883 tablet_bootstrap.cc:492] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 91c53a144b0242e491f9e8b8aaef3f02: Bootstrap starting.
I20260926 03:32:02.537433 7882 tablet_bootstrap.cc:654] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P b61e6b996c96424aa56ab30a31376891: Neither blocks nor log segments found. Creating new log.
I20260926 03:32:02.538355 7883 tablet_bootstrap.cc:654] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 91c53a144b0242e491f9e8b8aaef3f02: Neither blocks nor log segments found. Creating new log.
I20260926 03:32:02.539498 7882 tablet_bootstrap.cc:492] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P b61e6b996c96424aa56ab30a31376891: No bootstrap required, opened a new log
I20260926 03:32:02.539646 7882 ts_tablet_manager.cc:1397] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P b61e6b996c96424aa56ab30a31376891: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 03:32:02.540307 7882 raft_consensus.cc:348] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P b61e6b996c96424aa56ab30a31376891 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91c53a144b0242e491f9e8b8aaef3f02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36319 } } peers { permanent_uuid: "6b0027955ae14788b8afa531cbfd18e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41735 } } peers { permanent_uuid: "b61e6b996c96424aa56ab30a31376891" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34011 } }
I20260926 03:32:02.540544 7882 raft_consensus.cc:374] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P b61e6b996c96424aa56ab30a31376891 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:32:02.540664 7882 raft_consensus.cc:729] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P b61e6b996c96424aa56ab30a31376891 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b61e6b996c96424aa56ab30a31376891, State: Initialized, Role: FOLLOWER
I20260926 03:32:02.540794 7883 tablet_bootstrap.cc:492] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 91c53a144b0242e491f9e8b8aaef3f02: No bootstrap required, opened a new log
I20260926 03:32:02.540954 7883 ts_tablet_manager.cc:1397] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 91c53a144b0242e491f9e8b8aaef3f02: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 03:32:02.540937 7882 consensus_queue.cc:260] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P b61e6b996c96424aa56ab30a31376891 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91c53a144b0242e491f9e8b8aaef3f02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36319 } } peers { permanent_uuid: "6b0027955ae14788b8afa531cbfd18e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41735 } } peers { permanent_uuid: "b61e6b996c96424aa56ab30a31376891" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34011 } }
I20260926 03:32:02.541774 7882 ts_tablet_manager.cc:1428] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P b61e6b996c96424aa56ab30a31376891: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 03:32:02.545311 7881 tablet_bootstrap.cc:492] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 6b0027955ae14788b8afa531cbfd18e1: No bootstrap required, opened a new log
I20260926 03:32:02.545485 7881 ts_tablet_manager.cc:1397] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 6b0027955ae14788b8afa531cbfd18e1: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20260926 03:32:02.545300 7883 raft_consensus.cc:348] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 91c53a144b0242e491f9e8b8aaef3f02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91c53a144b0242e491f9e8b8aaef3f02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36319 } } peers { permanent_uuid: "6b0027955ae14788b8afa531cbfd18e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41735 } } peers { permanent_uuid: "b61e6b996c96424aa56ab30a31376891" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34011 } }
I20260926 03:32:02.545671 7883 raft_consensus.cc:374] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 91c53a144b0242e491f9e8b8aaef3f02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:32:02.545811 7883 raft_consensus.cc:729] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 91c53a144b0242e491f9e8b8aaef3f02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91c53a144b0242e491f9e8b8aaef3f02, State: Initialized, Role: FOLLOWER
I20260926 03:32:02.546104 7883 consensus_queue.cc:260] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 91c53a144b0242e491f9e8b8aaef3f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91c53a144b0242e491f9e8b8aaef3f02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36319 } } peers { permanent_uuid: "6b0027955ae14788b8afa531cbfd18e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41735 } } peers { permanent_uuid: "b61e6b996c96424aa56ab30a31376891" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34011 } }
I20260926 03:32:02.546277 7881 raft_consensus.cc:348] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 6b0027955ae14788b8afa531cbfd18e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91c53a144b0242e491f9e8b8aaef3f02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36319 } } peers { permanent_uuid: "6b0027955ae14788b8afa531cbfd18e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41735 } } peers { permanent_uuid: "b61e6b996c96424aa56ab30a31376891" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34011 } }
I20260926 03:32:02.546522 7881 raft_consensus.cc:374] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 6b0027955ae14788b8afa531cbfd18e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:32:02.546622 7881 raft_consensus.cc:729] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 6b0027955ae14788b8afa531cbfd18e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b0027955ae14788b8afa531cbfd18e1, State: Initialized, Role: FOLLOWER
I20260926 03:32:02.546906 7881 consensus_queue.cc:260] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 6b0027955ae14788b8afa531cbfd18e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91c53a144b0242e491f9e8b8aaef3f02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36319 } } peers { permanent_uuid: "6b0027955ae14788b8afa531cbfd18e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41735 } } peers { permanent_uuid: "b61e6b996c96424aa56ab30a31376891" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34011 } }
I20260926 03:32:02.547714 7881 ts_tablet_manager.cc:1428] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 6b0027955ae14788b8afa531cbfd18e1: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 03:32:02.548606 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.542031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46591 (local address 127.0.97.2:48319)
0926 03:32:02.548211 (+ 6180us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:02.548335 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6048,"thread_start_us":5939,"threads_started":1}
I20260926 03:32:02.551615 7645 heartbeater.cc:503] Master 127.0.97.60:36639 was elected leader, sending a full tablet report...
I20260926 03:32:02.554849 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.552364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46591 (local address 127.0.97.2:49407)
0926 03:32:02.554546 (+ 2182us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:02.554660 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2019}
I20260926 03:32:02.555665 7883 ts_tablet_manager.cc:1428] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 91c53a144b0242e491f9e8b8aaef3f02: Time spent starting tablet: real 0.015s user 0.006s sys 0.005s
I20260926 03:32:02.557307 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.555881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46591 (local address 127.0.97.3:40369)
0926 03:32:02.556929 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:02.557039 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":912,"thread_start_us":748,"threads_started":1}
I20260926 03:32:02.558585 7786 heartbeater.cc:503] Master 127.0.97.60:36639 was elected leader, sending a full tablet report...
I20260926 03:32:02.558671 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.558059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46591 (local address 127.0.97.3:59309)
0926 03:32:02.558459 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:02.558539 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279}
I20260926 03:32:02.559432 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.547944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46591 (local address 127.0.97.1:59893)
0926 03:32:02.559060 (+ 11116us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:02.559194 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10976,"threads_started":1}
I20260926 03:32:02.560665 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.560279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46591 (local address 127.0.97.1:46683)
0926 03:32:02.560424 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:02.560507 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 03:32:02.561517 7500 heartbeater.cc:503] Master 127.0.97.60:36639 was elected leader, sending a full tablet report...
I20260926 03:32:02.573563 7890 raft_consensus.cc:482] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 6b0027955ae14788b8afa531cbfd18e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:32:02.573822 7890 raft_consensus.cc:504] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 6b0027955ae14788b8afa531cbfd18e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91c53a144b0242e491f9e8b8aaef3f02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36319 } } peers { permanent_uuid: "6b0027955ae14788b8afa531cbfd18e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41735 } } peers { permanent_uuid: "b61e6b996c96424aa56ab30a31376891" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34011 } }
I20260926 03:32:02.575527 7890 leader_election.cc:288] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 6b0027955ae14788b8afa531cbfd18e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91c53a144b0242e491f9e8b8aaef3f02 (127.0.97.3:36319), b61e6b996c96424aa56ab30a31376891 (127.0.97.2:34011)
I20260926 03:32:02.582902 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.575908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34011 (local address 127.0.97.1:44939)
0926 03:32:02.576639 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:02.576656 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:02.576767 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:02.577153 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:02.577157 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:02.577175 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:32:02.577502 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:02.577514 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:02.578559 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:02.578564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:02.581148 (+ 2584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:02.581163 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:02.582294 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:02.582298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:02.582422 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:02.582446 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 03:32:02.582521 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 03:32:02.582612 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":184,"thread_start_us":466,"threads_started":1}
I20260926 03:32:02.582960 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.575891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44939 (local address 127.0.97.2:34011)
0926 03:32:02.576221 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:02.576228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:02.576753 (+ 525us) server_negotiation.cc:407] Connection header received
0926 03:32:02.576890 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:02.576895 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:02.576979 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:02.577087 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:32:02.577677 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:02.578359 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:02.581316 (+ 2957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:02.582136 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:02.582237 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:02.582313 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:02.582585 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 03:32:02.582723 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192}
I20260926 03:32:02.583840 7599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8e2cfc16e394c53b2e9e8bfcf6c9902" candidate_uuid: "6b0027955ae14788b8afa531cbfd18e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b61e6b996c96424aa56ab30a31376891" is_pre_election: true
I20260926 03:32:02.584285 7599 raft_consensus.cc:2454] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P b61e6b996c96424aa56ab30a31376891 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b0027955ae14788b8afa531cbfd18e1 in term 0.
I20260926 03:32:02.584483 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.575816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36319 (local address 127.0.97.1:34059)
0926 03:32:02.576143 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:02.576162 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:02.576366 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:02.576933 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:02.576936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:02.576970 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:32:02.577386 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:02.577399 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:02.578487 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:02.578495 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:02.581150 (+ 2655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:02.581163 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:02.583754 (+ 2591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:02.583761 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:02.583945 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:02.583981 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 03:32:02.584085 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 03:32:02.584209 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":178}
I20260926 03:32:02.584482 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.575822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34059 (local address 127.0.97.3:36319)
0926 03:32:02.576038 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:02.576045 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:02.576240 (+ 195us) server_negotiation.cc:407] Connection header received
0926 03:32:02.576552 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:02.576584 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:02.576667 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:02.576737 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:32:02.577619 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:02.578347 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:02.582381 (+ 4034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:02.583553 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:02.583649 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:02.583718 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:02.584149 (+ 431us) server_negotiation.cc:299] Negotiation successful
0926 03:32:02.584272 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 03:32:02.584928 7388 leader_election.cc:302] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 6b0027955ae14788b8afa531cbfd18e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b0027955ae14788b8afa531cbfd18e1, b61e6b996c96424aa56ab30a31376891; no voters:
I20260926 03:32:02.585114 7740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8e2cfc16e394c53b2e9e8bfcf6c9902" candidate_uuid: "6b0027955ae14788b8afa531cbfd18e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91c53a144b0242e491f9e8b8aaef3f02" is_pre_election: true
I20260926 03:32:02.585367 7890 raft_consensus.cc:2781] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 6b0027955ae14788b8afa531cbfd18e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:32:02.585381 7740 raft_consensus.cc:2454] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 91c53a144b0242e491f9e8b8aaef3f02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b0027955ae14788b8afa531cbfd18e1 in term 0.
I20260926 03:32:02.585500 7890 raft_consensus.cc:482] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 6b0027955ae14788b8afa531cbfd18e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:32:02.585620 7890 raft_consensus.cc:3037] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 6b0027955ae14788b8afa531cbfd18e1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:32:02.587743 7890 raft_consensus.cc:504] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 6b0027955ae14788b8afa531cbfd18e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91c53a144b0242e491f9e8b8aaef3f02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36319 } } peers { permanent_uuid: "6b0027955ae14788b8afa531cbfd18e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41735 } } peers { permanent_uuid: "b61e6b996c96424aa56ab30a31376891" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34011 } }
I20260926 03:32:02.588304 7890 leader_election.cc:288] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 6b0027955ae14788b8afa531cbfd18e1 [CANDIDATE]: Term 1 election: Requested vote from peers 91c53a144b0242e491f9e8b8aaef3f02 (127.0.97.3:36319), b61e6b996c96424aa56ab30a31376891 (127.0.97.2:34011)
I20260926 03:32:02.588758 7740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8e2cfc16e394c53b2e9e8bfcf6c9902" candidate_uuid: "6b0027955ae14788b8afa531cbfd18e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91c53a144b0242e491f9e8b8aaef3f02"
I20260926 03:32:02.588775 7599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8e2cfc16e394c53b2e9e8bfcf6c9902" candidate_uuid: "6b0027955ae14788b8afa531cbfd18e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b61e6b996c96424aa56ab30a31376891"
I20260926 03:32:02.588946 7740 raft_consensus.cc:3037] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 91c53a144b0242e491f9e8b8aaef3f02 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:32:02.588949 7599 raft_consensus.cc:3037] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P b61e6b996c96424aa56ab30a31376891 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:32:02.590889 7740 raft_consensus.cc:2454] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 91c53a144b0242e491f9e8b8aaef3f02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b0027955ae14788b8afa531cbfd18e1 in term 1.
I20260926 03:32:02.590898 7599 raft_consensus.cc:2454] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P b61e6b996c96424aa56ab30a31376891 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b0027955ae14788b8afa531cbfd18e1 in term 1.
I20260926 03:32:02.591287 7388 leader_election.cc:302] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 6b0027955ae14788b8afa531cbfd18e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b0027955ae14788b8afa531cbfd18e1, b61e6b996c96424aa56ab30a31376891; no voters:
I20260926 03:32:02.591574 7890 raft_consensus.cc:2781] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 6b0027955ae14788b8afa531cbfd18e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:32:02.592010 7890 raft_consensus.cc:686] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 6b0027955ae14788b8afa531cbfd18e1 [term 1 LEADER]: Becoming Leader. State: Replica: 6b0027955ae14788b8afa531cbfd18e1, State: Running, Role: LEADER
I20260926 03:32:02.592532 7890 consensus_queue.cc:237] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 6b0027955ae14788b8afa531cbfd18e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91c53a144b0242e491f9e8b8aaef3f02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36319 } } peers { permanent_uuid: "6b0027955ae14788b8afa531cbfd18e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41735 } } peers { permanent_uuid: "b61e6b996c96424aa56ab30a31376891" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34011 } }
I20260926 03:32:02.593302 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.592793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46591 (local address 127.0.97.1:43769)
0926 03:32:02.593037 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:02.593133 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260926 03:32:02.595966 7321 catalog_manager.cc:5495] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 6b0027955ae14788b8afa531cbfd18e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b0027955ae14788b8afa531cbfd18e1 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "6b0027955ae14788b8afa531cbfd18e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91c53a144b0242e491f9e8b8aaef3f02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b0027955ae14788b8afa531cbfd18e1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b61e6b996c96424aa56ab30a31376891" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34011 } health_report { overall_health: UNKNOWN } } }
I20260926 03:32:02.623428 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.618758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56240 (local address 127.0.97.1:41735)
0926 03:32:02.618951 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:02.618955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:02.618973 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:32:02.619182 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:02.619186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:02.619264 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:02.619399 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:32:02.619709 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:02.620372 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:02.621631 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:02.622043 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:02.622147 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:02.622465 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:32:02.622991 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:32:02.623094 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:02.623172 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 03:32:02.623234 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 03:32:02.627780 7740 raft_consensus.cc:1261] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 91c53a144b0242e491f9e8b8aaef3f02 [term 1 FOLLOWER]: Refusing update from remote peer 6b0027955ae14788b8afa531cbfd18e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:32:02.628306 7599 raft_consensus.cc:1261] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P b61e6b996c96424aa56ab30a31376891 [term 1 FOLLOWER]: Refusing update from remote peer 6b0027955ae14788b8afa531cbfd18e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:32:02.628533 7894 consensus_queue.cc:1035] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 6b0027955ae14788b8afa531cbfd18e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c53a144b0242e491f9e8b8aaef3f02" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:32:02.629065 7890 consensus_queue.cc:1035] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 6b0027955ae14788b8afa531cbfd18e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b61e6b996c96424aa56ab30a31376891" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:32:02.631639 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.631131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46591 (local address 127.0.97.2:35061)
0926 03:32:02.631356 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:02.631451 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260926 03:32:02.632009 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.631446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46591 (local address 127.0.97.3:37923)
0926 03:32:02.631687 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:02.631799 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260926 03:32:02.636093 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.635721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46591 (local address 127.0.97.3:60183)
0926 03:32:02.635855 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:02.635947 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 03:32:02.637517 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.636996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46591 (local address 127.0.97.1:35333)
0926 03:32:02.637269 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:02.637358 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
I20260926 03:32:02.639938 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.639417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46591 (local address 127.0.97.2:57109)
0926 03:32:02.639624 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:02.639737 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 03:32:02.635109 7898 mvcc.cc:205] Tried to move back new op lower bound from 7333451868676644864 to 7333451868543340544. Current Snapshot: MvccSnapshot[applied={T|T < 7333451868676644864}]
I20260926 03:32:02.651486 7320 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:32:02.651757 7320 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:32:02.657265 7320 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a56fb218c4e482fac40da5c0be63a91: Sending DeleteTablet for 3 replicas of tablet d8e2cfc16e394c53b2e9e8bfcf6c9902
I20260926 03:32:02.658344 7720 tablet_service.cc:1514] Processing DeleteTablet for tablet d8e2cfc16e394c53b2e9e8bfcf6c9902 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:32:02 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:34314
I20260926 03:32:02.658420 7579 tablet_service.cc:1514] Processing DeleteTablet for tablet d8e2cfc16e394c53b2e9e8bfcf6c9902 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:32:02 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:48080
I20260926 03:32:02.658787 7434 tablet_service.cc:1514] Processing DeleteTablet for tablet d8e2cfc16e394c53b2e9e8bfcf6c9902 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:32:02 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:56226
I20260926 03:32:02.659125 7907 tablet_replica.cc:330] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 91c53a144b0242e491f9e8b8aaef3f02: stopping tablet replica
I20260926 03:32:02.659709 7907 raft_consensus.cc:2229] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 91c53a144b0242e491f9e8b8aaef3f02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:32:02.660148 7907 raft_consensus.cc:2258] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 91c53a144b0242e491f9e8b8aaef3f02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:32:02.660856 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 7308
I20260926 03:32:02.664837 7907 ts_tablet_manager.cc:1905] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 91c53a144b0242e491f9e8b8aaef3f02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:32:02.669258 7909 tablet_replica.cc:330] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 6b0027955ae14788b8afa531cbfd18e1: stopping tablet replica
I20260926 03:32:02.669768 7908 tablet_replica.cc:330] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P b61e6b996c96424aa56ab30a31376891: stopping tablet replica
I20260926 03:32:02.669462 7909 raft_consensus.cc:2229] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 6b0027955ae14788b8afa531cbfd18e1 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:32:02.670550 7909 raft_consensus.cc:2258] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 6b0027955ae14788b8afa531cbfd18e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:32:02.670660 7908 raft_consensus.cc:2229] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P b61e6b996c96424aa56ab30a31376891 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:32:02.671679 7909 ts_tablet_manager.cc:1905] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 6b0027955ae14788b8afa531cbfd18e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:32:02.671984 7908 raft_consensus.cc:2258] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P b61e6b996c96424aa56ab30a31376891 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:32:02.675146 7907 ts_tablet_manager.cc:1918] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 91c53a144b0242e491f9e8b8aaef3f02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:32:02.676247 7908 ts_tablet_manager.cc:1905] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P b61e6b996c96424aa56ab30a31376891: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:32:02.679026 7909 ts_tablet_manager.cc:1918] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 6b0027955ae14788b8afa531cbfd18e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:32:02.679170 7909 log.cc:1196] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 6b0027955ae14788b8afa531cbfd18e1: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-0/wal/wals/d8e2cfc16e394c53b2e9e8bfcf6c9902
I20260926 03:32:02.679319 7907 log.cc:1196] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 91c53a144b0242e491f9e8b8aaef3f02: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-2/wal/wals/d8e2cfc16e394c53b2e9e8bfcf6c9902
I20260926 03:32:02.679697 7909 ts_tablet_manager.cc:1939] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 6b0027955ae14788b8afa531cbfd18e1: Deleting consensus metadata
I20260926 03:32:02.680994 7304 catalog_manager.cc:4841] TS 6b0027955ae14788b8afa531cbfd18e1 (127.0.97.1:41735): tablet d8e2cfc16e394c53b2e9e8bfcf6c9902 (table test-table [id=2d4a4801a91d4ebb8c894c379f13b4c2]) successfully deleted
I20260926 03:32:02.681327 7907 ts_tablet_manager.cc:1939] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P 91c53a144b0242e491f9e8b8aaef3f02: Deleting consensus metadata
I20260926 03:32:02.682542 7307 catalog_manager.cc:4841] TS 91c53a144b0242e491f9e8b8aaef3f02 (127.0.97.3:36319): tablet d8e2cfc16e394c53b2e9e8bfcf6c9902 (table test-table [id=2d4a4801a91d4ebb8c894c379f13b4c2]) successfully deleted
I20260926 03:32:02.684921 7908 ts_tablet_manager.cc:1918] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P b61e6b996c96424aa56ab30a31376891: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:32:02.685079 7908 log.cc:1196] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P b61e6b996c96424aa56ab30a31376891: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790393463219689-388-0/minicluster-data/ts-1/wal/wals/d8e2cfc16e394c53b2e9e8bfcf6c9902
I20260926 03:32:02.685910 7908 ts_tablet_manager.cc:1939] T d8e2cfc16e394c53b2e9e8bfcf6c9902 P b61e6b996c96424aa56ab30a31376891: Deleting consensus metadata
I20260926 03:32:02.686932 7305 catalog_manager.cc:4841] TS b61e6b996c96424aa56ab30a31376891 (127.0.97.2:34011): tablet d8e2cfc16e394c53b2e9e8bfcf6c9902 (table test-table [id=2d4a4801a91d4ebb8c894c379f13b4c2]) successfully deleted
I20260926 03:32:02.691273 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.662173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56244 (local address 127.0.97.1:41735)
0926 03:32:02.663648 (+ 1475us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:02.663662 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:02.673837 (+ 10175us) server_negotiation.cc:407] Connection header received
0926 03:32:02.674282 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:02.674288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:02.674369 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:02.674464 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:02.676926 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:02.678187 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:02.680067 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:02.680759 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:02.680877 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:02.684415 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:02.684430 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:02.684452 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:02.684490 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:02.688266 (+ 3776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:02.689094 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:02.689101 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:02.689103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:02.689273 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:02.689711 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:02.689714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:02.689716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:02.690185 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:02.690480 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:02.690783 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 03:32:02.690944 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1316}
I20260926 03:32:02.691954 7494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:32:02.805922 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 7308
I20260926 03:32:02.824827 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 7511
I20260926 03:32:02.839105 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.825629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48092 (local address 127.0.97.2:34011)
0926 03:32:02.825794 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:02.825798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:02.825887 (+ 89us) server_negotiation.cc:407] Connection header received
0926 03:32:02.826075 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:02.826081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:02.826157 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:02.826292 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:02.827537 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:02.828241 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:02.829819 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:02.830229 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:02.830366 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:02.832969 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:02.832983 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:02.833000 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:02.833035 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:02.836224 (+ 3189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:02.837138 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:02.837145 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:02.837147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:02.837276 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:02.837629 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:02.837632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:02.837634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:02.838099 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:02.838348 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:02.838605 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 03:32:02.838790 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 03:32:02.839922 7639 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:32:02.903759 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.903330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46591 (local address 127.0.0.1:43162)
0926 03:32:02.903508 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:02.903598 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 03:32:02.904796 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.904411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46591 (local address 127.0.0.1:43170)
0926 03:32:02.904535 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:02.904667 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 03:32:02.965143 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 7511
I20260926 03:32:02.983580 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 7657
I20260926 03:32:02.996186 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:02.984346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34320 (local address 127.0.97.3:36319)
0926 03:32:02.984507 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:02.984511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:02.984528 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:32:02.984752 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:02.984756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:02.984816 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:02.984915 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:02.986174 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:02.986860 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:02.987971 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:02.988360 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:02.988447 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:02.990824 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:02.990836 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:02.990853 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:02.990884 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:02.993680 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:02.994270 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:02.994276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:02.994278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:02.994389 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:02.994693 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:02.994696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:02.994698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:02.995052 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:02.995273 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:02.995643 (+ 370us) server_negotiation.cc:299] Negotiation successful
0926 03:32:02.995792 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 03:32:02.996999 7780 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:32:03.090199 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 7657
I20260926 03:32:03.108137 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 6997
I20260926 03:32:03.109380 7163 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:32:03.226238 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 6997
I20260926 03:32:03.247220 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 7096
I20260926 03:32:03.248286 7351 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:32:03.367223 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:03.366713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42461 (local address 127.0.0.1:44720)
0926 03:32:03.366913 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:03.367010 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 03:32:03.367264 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 7096
2026-09-26T03:32:03Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9602 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:32:03 dist-test-slave-dt7p krb5kdc[7920](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:32:03 dist-test-slave-dt7p krb5kdc[7920](info): set up 2 sockets
Sep 26 03:32:03 dist-test-slave-dt7p krb5kdc[7920](info): commencing operation
krb5kdc: starting...
W20260926 03:32:04.485769 388 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.004s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:32:04 dist-test-slave-dt7p krb5kdc[7920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:32:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:32:04Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260926 03:32:04.708456 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33533
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:32:05.032617 7933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:32:05.033008 7933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:32:05.033178 7933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:32:05.040473 7933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:32:05.040666 7933 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 03:32:05.040759 7933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:32:05.040853 7933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:32:05.040990 7933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:32:05.041102 7933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:32:05.051126 7933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33533
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:32:05.052492 7933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:32:05.053813 7933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:32:05.065598 7939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:05.065641 7938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:05.065799 7941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:32:05.066593 7933 server_base.cc:1034] running on GCE node
I20260926 03:32:05.067376 7933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:32:05.068928 7933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:32:05.070205 7933 hybrid_clock.cc:648] HybridClock initialized: now 1790393525070155 us; error 51 us; skew 500 ppm
I20260926 03:32:05.070561 7933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:32:05 dist-test-slave-dt7p krb5kdc[7920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393525, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:32:05.073818 7933 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260926 03:32:05.075289 7933 webserver.cc:460] Webserver started at http://127.0.97.62:34555/ using document root <none> and password file <none>
I20260926 03:32:05.076013 7933 fs_manager.cc:359] Metadata directory not provided
I20260926 03:32:05.076155 7933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:32:05.076501 7933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:32:05.079697 7933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "dbe96762262e460388583b79167d2c1d"
format_stamp: "Formatted at 2026-09-26 03:32:05 on dist-test-slave-dt7p"
server_key: "2fd780bd9917cc1f314157f59e613440"
server_key_iv: "fc49c80b0954c0ec884ba456e946bbf7"
server_key_version: "encryptionkey@0"
I20260926 03:32:05.080502 7933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "dbe96762262e460388583b79167d2c1d"
format_stamp: "Formatted at 2026-09-26 03:32:05 on dist-test-slave-dt7p"
server_key: "2fd780bd9917cc1f314157f59e613440"
server_key_iv: "fc49c80b0954c0ec884ba456e946bbf7"
server_key_version: "encryptionkey@0"
I20260926 03:32:05.085860 7933 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 03:32:05.089696 7948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:05.091074 7933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:32:05.091282 7933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "dbe96762262e460388583b79167d2c1d"
format_stamp: "Formatted at 2026-09-26 03:32:05 on dist-test-slave-dt7p"
server_key: "2fd780bd9917cc1f314157f59e613440"
server_key_iv: "fc49c80b0954c0ec884ba456e946bbf7"
server_key_version: "encryptionkey@0"
I20260926 03:32:05.091508 7933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:32:05.109663 7933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:32:05.113708 7933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:32:05.113986 7933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:32:05.135226 7933 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33533
I20260926 03:32:05.135245 8009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33533 every 8 connection(s)
I20260926 03:32:05.136682 7933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:32:05.139727 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 7933
I20260926 03:32:05.140009 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:32:05.140532 8010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:32:05.140733 388 external_mini_cluster.cc:1428] Setting key 05fdaa97b33de6351b6b7ddfb44b1e6a
I20260926 03:32:05.155510 8010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbe96762262e460388583b79167d2c1d: Bootstrap starting.
Sep 26 03:32:05 dist-test-slave-dt7p krb5kdc[7920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:32:05.158829 8010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbe96762262e460388583b79167d2c1d: Neither blocks nor log segments found. Creating new log.
I20260926 03:32:05.160234 8010 log.cc:824] T 00000000000000000000000000000000 P dbe96762262e460388583b79167d2c1d: Log is configured to *not* fsync() on all Append() calls
I20260926 03:32:05.163691 8010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbe96762262e460388583b79167d2c1d: No bootstrap required, opened a new log
I20260926 03:32:05.165642 8013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:05.142682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37054 (local address 127.0.97.62:33533)
0926 03:32:05.143541 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:05.143553 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:05.143592 (+ 39us) server_negotiation.cc:407] Connection header received
0926 03:32:05.145544 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:05.145568 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:05.145997 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:05.146376 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:05.148242 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:05.149678 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:05.151417 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:05.152167 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:05.152331 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:05.157219 (+ 4888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:05.157227 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:05.157242 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:05.157280 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:05.161164 (+ 3884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:05.161947 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:05.161955 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:05.161962 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:05.162167 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:05.162535 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:05.162539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:05.162541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:05.163209 (+ 668us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:05.163514 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:05.163864 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 03:32:05.164304 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":401,"threads_started":1}
I20260926 03:32:05.172153 8010 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbe96762262e460388583b79167d2c1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe96762262e460388583b79167d2c1d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33533 } }
I20260926 03:32:05.172456 8010 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbe96762262e460388583b79167d2c1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:32:05.172528 8010 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbe96762262e460388583b79167d2c1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbe96762262e460388583b79167d2c1d, State: Initialized, Role: FOLLOWER
I20260926 03:32:05.173110 8010 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbe96762262e460388583b79167d2c1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe96762262e460388583b79167d2c1d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33533 } }
I20260926 03:32:05.173305 8010 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbe96762262e460388583b79167d2c1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:32:05.173394 8010 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbe96762262e460388583b79167d2c1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:32:05.173512 8010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbe96762262e460388583b79167d2c1d [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:32:05.175522 8010 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbe96762262e460388583b79167d2c1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe96762262e460388583b79167d2c1d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33533 } }
I20260926 03:32:05.175987 8010 leader_election.cc:302] T 00000000000000000000000000000000 P dbe96762262e460388583b79167d2c1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbe96762262e460388583b79167d2c1d; no voters:
I20260926 03:32:05.176527 8010 leader_election.cc:288] T 00000000000000000000000000000000 P dbe96762262e460388583b79167d2c1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:32:05.176743 8015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbe96762262e460388583b79167d2c1d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:32:05.177968 8015 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbe96762262e460388583b79167d2c1d [term 1 LEADER]: Becoming Leader. State: Replica: dbe96762262e460388583b79167d2c1d, State: Running, Role: LEADER
I20260926 03:32:05.178498 8015 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbe96762262e460388583b79167d2c1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe96762262e460388583b79167d2c1d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33533 } }
I20260926 03:32:05.179003 8010 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbe96762262e460388583b79167d2c1d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:32:05.181523 8017 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbe96762262e460388583b79167d2c1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbe96762262e460388583b79167d2c1d. Latest consensus state: current_term: 1 leader_uuid: "dbe96762262e460388583b79167d2c1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe96762262e460388583b79167d2c1d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33533 } } }
I20260926 03:32:05.182022 8017 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbe96762262e460388583b79167d2c1d [sys.catalog]: This master's current role is: LEADER
I20260926 03:32:05.181656 8016 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbe96762262e460388583b79167d2c1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbe96762262e460388583b79167d2c1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe96762262e460388583b79167d2c1d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33533 } } }
I20260926 03:32:05.182451 8016 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbe96762262e460388583b79167d2c1d [sys.catalog]: This master's current role is: LEADER
I20260926 03:32:05.190546 8023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:32:05.196209 8023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:32:05.205783 8023 catalog_manager.cc:1347] Generated new cluster ID: 190e351b33d2487f875a714957d994be
I20260926 03:32:05.205945 8023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:32:05.217638 8023 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:32:05.219058 8023 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:32:05.231102 8023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbe96762262e460388583b79167d2c1d: Generated new TSK 0
I20260926 03:32:05.232021 8023 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260926 03:32:05.293622 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33533
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:32:05.646500 8036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:32:05.646831 8036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:32:05.647017 8036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:32:05.654186 8036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:32:05.654349 8036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:32:05.654520 8036 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:32:05.654675 8036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:32:05.664610 8036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:33533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:32:05.665953 8036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:32:05.667243 8036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:32:05.680446 8041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:05.680317 8044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:05.680322 8042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:32:05.681794 8036 server_base.cc:1034] running on GCE node
I20260926 03:32:05.682446 8036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:32:05.683385 8036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:32:05.684677 8036 hybrid_clock.cc:648] HybridClock initialized: now 1790393525684605 us; error 59 us; skew 500 ppm
I20260926 03:32:05.684994 8036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:32:05 dist-test-slave-dt7p krb5kdc[7920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393525, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:32:05.688019 8036 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260926 03:32:05.689627 8036 webserver.cc:460] Webserver started at http://127.0.97.1:35631/ using document root <none> and password file <none>
I20260926 03:32:05.690378 8036 fs_manager.cc:359] Metadata directory not provided
I20260926 03:32:05.690521 8036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:32:05.690861 8036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:32:05.693768 8036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-0/data/instance:
uuid: "b2b9ca83586547e18d54d7eb8eed7b76"
format_stamp: "Formatted at 2026-09-26 03:32:05 on dist-test-slave-dt7p"
server_key: "2ee120efcc1ede76c9bc2dd7f1a54e14"
server_key_iv: "6aaa3152a68e0048b31c27ba090dec81"
server_key_version: "encryptionkey@0"
I20260926 03:32:05.694538 8036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b2b9ca83586547e18d54d7eb8eed7b76"
format_stamp: "Formatted at 2026-09-26 03:32:05 on dist-test-slave-dt7p"
server_key: "2ee120efcc1ede76c9bc2dd7f1a54e14"
server_key_iv: "6aaa3152a68e0048b31c27ba090dec81"
server_key_version: "encryptionkey@0"
I20260926 03:32:05.699791 8036 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 03:32:05.703341 8051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:05.704634 8036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:32:05.704804 8036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-0/wal
uuid: "b2b9ca83586547e18d54d7eb8eed7b76"
format_stamp: "Formatted at 2026-09-26 03:32:05 on dist-test-slave-dt7p"
server_key: "2ee120efcc1ede76c9bc2dd7f1a54e14"
server_key_iv: "6aaa3152a68e0048b31c27ba090dec81"
server_key_version: "encryptionkey@0"
I20260926 03:32:05.705039 8036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:32:05.716411 8036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:32:05.720170 8036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:32:05.720440 8036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:32:05.723150 8036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:32:05.723263 8036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:05.723466 8036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:32:05.723553 8036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:32:05 dist-test-slave-dt7p krb5kdc[7920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393525, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:32:05.746901 8061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:05.724201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33533 (local address 127.0.97.1:54479)
0926 03:32:05.725088 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:05.725135 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:05.726964 (+ 1829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:05.727690 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:05.727697 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:05.728426 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:05.729342 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:05.729374 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:05.730852 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:05.730862 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:05.732272 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:05.732283 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:05.733209 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:05.733222 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:05.733551 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:05.734460 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:05.734486 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:05.737940 (+ 3454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:32:05.741483 (+ 3543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:32:05.741495 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:32:05.741512 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:32:05.742186 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:32:05.742689 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:32:05.742696 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:32:05.742704 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:32:05.742972 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:32:05.743673 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:32:05.743684 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:32:05.744221 (+ 537us) client_negotiation.cc:769] Sending connection context
0926 03:32:05.744673 (+ 452us) client_negotiation.cc:240] Negotiation successful
0926 03:32:05.745290 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":382,"threads_started":1}
I20260926 03:32:05.747886 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:05.723558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54479 (local address 127.0.97.62:33533)
0926 03:32:05.724107 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:05.724114 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:05.725275 (+ 1161us) server_negotiation.cc:407] Connection header received
0926 03:32:05.727240 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:05.727245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:05.727336 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:05.727463 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:05.729602 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:05.730685 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:05.732425 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:05.732910 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:05.733077 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:05.738173 (+ 5096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:05.738179 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:05.738184 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:05.738225 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:05.741335 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:05.742366 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:05.742372 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:05.742375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:05.742530 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:05.743105 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:05.743110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:05.743114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:05.743530 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:05.743696 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:05.747480 (+ 3784us) server_negotiation.cc:299] Negotiation successful
0926 03:32:05.747650 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":259,"threads_started":1}
I20260926 03:32:05.770068 8126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:05.755083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33533 (local address 127.0.97.1:45351)
0926 03:32:05.755764 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:05.755782 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:05.755966 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:05.756318 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:05.756323 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:05.757204 (+ 881us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:05.757761 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:05.757780 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:05.759470 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:05.759478 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:05.761132 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:05.761148 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:05.762467 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:05.762477 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:05.762697 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:05.763438 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:05.763457 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:05.764463 (+ 1006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:32:05.767512 (+ 3049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:32:05.767519 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:32:05.767523 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:32:05.767894 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:32:05.768276 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:32:05.768281 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:32:05.768285 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:32:05.768412 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:32:05.769002 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:32:05.769007 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:32:05.769208 (+ 201us) client_negotiation.cc:769] Sending connection context
0926 03:32:05.769359 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 03:32:05.769731 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":388,"threads_started":1}
I20260926 03:32:05.770368 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:05.755042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45351 (local address 127.0.97.62:33533)
0926 03:32:05.755205 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:05.755210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:05.755882 (+ 672us) server_negotiation.cc:407] Connection header received
0926 03:32:05.756105 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:05.756115 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:05.756179 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:05.756282 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:05.757952 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:05.759302 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:05.761370 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:05.762045 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:05.762183 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:05.764697 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:05.764702 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:05.764705 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:05.764735 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:05.767278 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:05.768030 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:05.768036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:05.768040 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:05.768161 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:05.768528 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:05.768533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:05.768536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:05.768876 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:05.769026 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:05.769557 (+ 531us) server_negotiation.cc:299] Negotiation successful
0926 03:32:05.769864 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 03:32:05.776662 8036 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46445
I20260926 03:32:05.776669 8171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46445 every 8 connection(s)
I20260926 03:32:05.777840 8036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:32:05.779870 8172 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33533
I20260926 03:32:05.780082 8172 heartbeater.cc:464] Registering TS with master...
I20260926 03:32:05.780791 8172 heartbeater.cc:511] Master 127.0.97.62:33533 requested a full tablet report, sending...
I20260926 03:32:05.782440 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 8036
I20260926 03:32:05.782676 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:32:05.782932 7965 ts_manager.cc:194] Registered new tserver with Master: b2b9ca83586547e18d54d7eb8eed7b76 (127.0.97.1:46445)
I20260926 03:32:05.783491 388 external_mini_cluster.cc:1428] Setting key 04cb0ac5e634f45ce39607fddb8f643e
I20260926 03:32:05.784844 7965 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54479
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260926 03:32:05.832290 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33533
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:32:06.155648 8176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:32:06.155961 8176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:32:06.156136 8176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:32:06.163898 8176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:32:06.164053 8176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:32:06.164206 8176 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:32:06.164348 8176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:32:06.174161 8176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:33533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:32:06.175474 8176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:32:06.176815 8176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:32:06.190387 8181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:06.190477 8184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:06.190438 8182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:32:06.191224 8176 server_base.cc:1034] running on GCE node
I20260926 03:32:06.191859 8176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:32:06.192865 8176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:32:06.194093 8176 hybrid_clock.cc:648] HybridClock initialized: now 1790393526194021 us; error 44 us; skew 500 ppm
I20260926 03:32:06.194360 8176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:32:06 dist-test-slave-dt7p krb5kdc[7920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393526, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:32:06.197878 8176 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260926 03:32:06.199355 8176 webserver.cc:460] Webserver started at http://127.0.97.2:36769/ using document root <none> and password file <none>
I20260926 03:32:06.200021 8176 fs_manager.cc:359] Metadata directory not provided
I20260926 03:32:06.200109 8176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:32:06.200366 8176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:32:06.203334 8176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-1/data/instance:
uuid: "7b178dcd80ae46949a1793f90f4a5cf9"
format_stamp: "Formatted at 2026-09-26 03:32:06 on dist-test-slave-dt7p"
server_key: "bb260ae3b3b22842b5bd47bff0736e18"
server_key_iv: "4b08d7296ba678a5c2de608d6806d3a9"
server_key_version: "encryptionkey@0"
I20260926 03:32:06.204103 8176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7b178dcd80ae46949a1793f90f4a5cf9"
format_stamp: "Formatted at 2026-09-26 03:32:06 on dist-test-slave-dt7p"
server_key: "bb260ae3b3b22842b5bd47bff0736e18"
server_key_iv: "4b08d7296ba678a5c2de608d6806d3a9"
server_key_version: "encryptionkey@0"
I20260926 03:32:06.209748 8176 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 03:32:06.213410 8191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:06.214709 8176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:32:06.214902 8176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-1/wal
uuid: "7b178dcd80ae46949a1793f90f4a5cf9"
format_stamp: "Formatted at 2026-09-26 03:32:06 on dist-test-slave-dt7p"
server_key: "bb260ae3b3b22842b5bd47bff0736e18"
server_key_iv: "4b08d7296ba678a5c2de608d6806d3a9"
server_key_version: "encryptionkey@0"
I20260926 03:32:06.215133 8176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:32:06.250605 8176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:32:06.255383 8176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:32:06.255631 8176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:32:06.258280 8176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:32:06.258406 8176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:06.258538 8176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:32:06.258613 8176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:32:06 dist-test-slave-dt7p krb5kdc[7920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393526, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:32:06.284952 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:06.259110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41535 (local address 127.0.97.62:33533)
0926 03:32:06.259303 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:06.259310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:06.261020 (+ 1710us) server_negotiation.cc:407] Connection header received
0926 03:32:06.263662 (+ 2642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:06.263668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:06.263762 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:06.263883 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:06.266503 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:06.267820 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:06.270325 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:06.271124 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:06.271275 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:06.277143 (+ 5868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:06.277154 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:06.277162 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:06.277220 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:06.281173 (+ 3953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:06.282176 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:06.282182 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:06.282186 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:06.282364 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:06.282979 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:06.282987 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:06.282993 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:06.283370 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:06.283534 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:06.284296 (+ 762us) server_negotiation.cc:299] Negotiation successful
0926 03:32:06.284533 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 03:32:06.286566 8200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:06.259990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33533 (local address 127.0.97.2:41535)
0926 03:32:06.260863 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:06.260904 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:06.263319 (+ 2415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:06.264178 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:06.264191 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:06.265327 (+ 1136us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:06.266221 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:06.266244 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:06.268033 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:06.268044 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:06.270139 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:06.270161 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:06.271664 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:06.271673 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:06.272057 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:06.273027 (+ 970us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:06.273057 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:06.276729 (+ 3672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:32:06.281330 (+ 4601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:32:06.281343 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:32:06.281364 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:32:06.282032 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:32:06.282556 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:32:06.282564 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:32:06.282571 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:32:06.282787 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:32:06.283524 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:32:06.283532 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:32:06.283905 (+ 373us) client_negotiation.cc:769] Sending connection context
0926 03:32:06.284205 (+ 300us) client_negotiation.cc:240] Negotiation successful
0926 03:32:06.284820 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":324,"threads_started":1}
I20260926 03:32:06.300244 8176 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35309
I20260926 03:32:06.300266 8310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35309 every 8 connection(s)
I20260926 03:32:06.301774 8176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:32:06.303086 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 8176
I20260926 03:32:06.303293 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:32:06.303714 388 external_mini_cluster.cc:1428] Setting key 910c20c9999802689f976d95da594432
I20260926 03:32:06.306895 8311 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33533
I20260926 03:32:06.307137 8311 heartbeater.cc:464] Registering TS with master...
I20260926 03:32:06.307899 8311 heartbeater.cc:511] Master 127.0.97.62:33533 requested a full tablet report, sending...
I20260926 03:32:06.309504 7965 ts_manager.cc:194] Registered new tserver with Master: 7b178dcd80ae46949a1793f90f4a5cf9 (127.0.97.2:35309)
I20260926 03:32:06.310746 7965 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41535
I20260926 03:32:06.311298 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:06.294126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42617 (local address 127.0.97.62:33533)
0926 03:32:06.294357 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:06.294365 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:06.295572 (+ 1207us) server_negotiation.cc:407] Connection header received
0926 03:32:06.295692 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:06.295701 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:06.295796 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:06.295900 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:06.297737 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:06.298670 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:06.300956 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:06.301368 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:06.301494 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:06.305164 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:06.305169 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:06.305172 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:06.305201 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:06.308279 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:06.309242 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:06.309246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:06.309248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:06.309355 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:06.309791 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:06.309796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:06.309799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:06.310150 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:06.310302 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:06.310679 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 03:32:06.310974 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":71}
I20260926 03:32:06.311355 8296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:06.294135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33533 (local address 127.0.97.2:42617)
0926 03:32:06.295464 (+ 1329us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:06.295479 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:06.295590 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:06.296180 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:06.296189 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:06.296953 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:06.297547 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:06.297567 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:06.298840 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:06.298849 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:06.300800 (+ 1951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:06.300814 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:06.301566 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:06.301572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:06.301772 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:06.302500 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:06.302534 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:06.304978 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:32:06.308482 (+ 3504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:32:06.308486 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:32:06.308488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:32:06.309097 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:32:06.309513 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:32:06.309519 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:32:06.309522 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:32:06.309668 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:32:06.310267 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:32:06.310272 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:32:06.310441 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 03:32:06.310575 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 03:32:06.310885 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1219,"thread_start_us":819,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260926 03:32:06.356177 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33533
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:32:06.687474 8315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:32:06.687760 8315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:32:06.687856 8315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:32:06.695559 8315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:32:06.695675 8315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:32:06.695791 8315 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:32:06.695904 8315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:32:06.705834 8315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:33533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:32:06.707130 8315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:32:06.708352 8315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:32:06.721704 8323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:06.721669 8321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:06.721822 8320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:32:06.722604 8315 server_base.cc:1034] running on GCE node
I20260926 03:32:06.723208 8315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:32:06.724092 8315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:32:06.725349 8315 hybrid_clock.cc:648] HybridClock initialized: now 1790393526725290 us; error 57 us; skew 500 ppm
I20260926 03:32:06.725670 8315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:32:06 dist-test-slave-dt7p krb5kdc[7920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393526, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:32:06.729036 8315 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260926 03:32:06.730428 8315 webserver.cc:460] Webserver started at http://127.0.97.3:34657/ using document root <none> and password file <none>
I20260926 03:32:06.731129 8315 fs_manager.cc:359] Metadata directory not provided
I20260926 03:32:06.731263 8315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:32:06.731591 8315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:32:06.734444 8315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-2/data/instance:
uuid: "261504269b4c4c7e9d96e8fd93cff7dc"
format_stamp: "Formatted at 2026-09-26 03:32:06 on dist-test-slave-dt7p"
server_key: "3bf1f98bd600eee58dc636ecf49ad0c1"
server_key_iv: "1c771776c1d19de3a7c7c4bb344cf569"
server_key_version: "encryptionkey@0"
I20260926 03:32:06.735240 8315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-2/wal/instance:
uuid: "261504269b4c4c7e9d96e8fd93cff7dc"
format_stamp: "Formatted at 2026-09-26 03:32:06 on dist-test-slave-dt7p"
server_key: "3bf1f98bd600eee58dc636ecf49ad0c1"
server_key_iv: "1c771776c1d19de3a7c7c4bb344cf569"
server_key_version: "encryptionkey@0"
I20260926 03:32:06.740240 8315 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:32:06.743494 8330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:06.744731 8315 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 03:32:06.744915 8315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-2/wal
uuid: "261504269b4c4c7e9d96e8fd93cff7dc"
format_stamp: "Formatted at 2026-09-26 03:32:06 on dist-test-slave-dt7p"
server_key: "3bf1f98bd600eee58dc636ecf49ad0c1"
server_key_iv: "1c771776c1d19de3a7c7c4bb344cf569"
server_key_version: "encryptionkey@0"
I20260926 03:32:06.745136 8315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:32:06.770885 8315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:32:06.775308 8315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:32:06.775605 8315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:32:06.778957 8315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:32:06.779114 8315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:06.779281 8315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:32:06.779361 8315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:06.787484 8172 heartbeater.cc:503] Master 127.0.97.62:33533 was elected leader, sending a full tablet report...
Sep 26 03:32:06 dist-test-slave-dt7p krb5kdc[7920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393526, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:32:06.802161 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:06.779204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40921 (local address 127.0.97.62:33533)
0926 03:32:06.779398 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:06.779405 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:06.780976 (+ 1571us) server_negotiation.cc:407] Connection header received
0926 03:32:06.783356 (+ 2380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:06.783363 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:06.783453 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:06.783575 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:06.785183 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:06.786310 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:06.789542 (+ 3232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:06.790223 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:06.790351 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:06.795269 (+ 4918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:06.795277 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:06.795283 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:06.795323 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:06.798492 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:06.799539 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:06.799545 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:06.799549 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:06.799684 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:06.800162 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:06.800171 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:06.800174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:06.800456 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:06.800639 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:06.801594 (+ 955us) server_negotiation.cc:299] Negotiation successful
0926 03:32:06.801812 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:32:06.803210 8339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:06.779796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33533 (local address 127.0.97.3:40921)
0926 03:32:06.780798 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:06.780850 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:06.783061 (+ 2211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:06.783768 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:06.783778 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:06.784294 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:06.785007 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:06.785025 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:06.786538 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:06.786544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:06.789366 (+ 2822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:06.789386 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:06.790644 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:06.790654 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:06.791026 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:06.791838 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:06.791874 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:06.795000 (+ 3126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:32:06.798650 (+ 3650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:32:06.798659 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:32:06.798675 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:32:06.799398 (+ 723us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:32:06.799800 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:32:06.799806 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:32:06.799809 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:32:06.800008 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:32:06.800632 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:32:06.800639 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:32:06.801096 (+ 457us) client_negotiation.cc:769] Sending connection context
0926 03:32:06.801472 (+ 376us) client_negotiation.cc:240] Negotiation successful
0926 03:32:06.802025 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":417,"threads_started":1}
I20260926 03:32:06.829391 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:06.810731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50421 (local address 127.0.97.62:33533)
0926 03:32:06.810906 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:06.810914 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:06.812490 (+ 1576us) server_negotiation.cc:407] Connection header received
0926 03:32:06.812713 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:06.812720 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:06.812815 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:06.812902 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:06.814529 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:06.815258 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:06.817889 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:06.818678 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:06.818793 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:06.821923 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:06.821932 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:06.821940 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:06.821980 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:06.825674 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:06.826771 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:06.826779 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:06.826784 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:06.826968 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:06.827563 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:06.827569 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:06.827574 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:06.828018 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:06.828171 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:06.828786 (+ 615us) server_negotiation.cc:299] Negotiation successful
0926 03:32:06.829021 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 03:32:06.830708 8392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:06.811168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33533 (local address 127.0.97.3:50421)
0926 03:32:06.812344 (+ 1176us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:06.812367 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:06.812582 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:06.812978 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:06.812983 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:06.813832 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:06.814360 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:06.814378 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:06.815412 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:06.815422 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:06.817680 (+ 2258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:06.817696 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:06.819141 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:06.819147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:06.819408 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:06.820199 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:06.820229 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:06.821667 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:32:06.825848 (+ 4181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:32:06.825856 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:32:06.825863 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:32:06.826592 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:32:06.827146 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:32:06.827154 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:32:06.827159 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:32:06.827407 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:32:06.828233 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:32:06.828236 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:32:06.828408 (+ 172us) client_negotiation.cc:769] Sending connection context
0926 03:32:06.829602 (+ 1194us) client_negotiation.cc:240] Negotiation successful
0926 03:32:06.830137 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":998,"thread_start_us":687,"threads_started":1}
I20260926 03:32:06.832968 8315 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43645
I20260926 03:32:06.832988 8449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43645 every 8 connection(s)
I20260926 03:32:06.834271 8315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:32:06.836295 8450 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33533
I20260926 03:32:06.836524 8450 heartbeater.cc:464] Registering TS with master...
I20260926 03:32:06.837157 8450 heartbeater.cc:511] Master 127.0.97.62:33533 requested a full tablet report, sending...
I20260926 03:32:06.838436 7965 ts_manager.cc:194] Registered new tserver with Master: 261504269b4c4c7e9d96e8fd93cff7dc (127.0.97.3:43645)
I20260926 03:32:06.838457 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 8315
I20260926 03:32:06.838692 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:32:06.839172 388 external_mini_cluster.cc:1428] Setting key 11dbd3a1fc2ac4cfa7ec1cc6deb0faeb
I20260926 03:32:06.839442 7965 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40921
I20260926 03:32:06.842059 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 03:32:06 dist-test-slave-dt7p krb5kdc[7920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393526, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 03:32:06 dist-test-slave-dt7p krb5kdc[7920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:06 dist-test-slave-dt7p krb5kdc[7920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 03:32:06.890300 8059 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:06.879211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37066 (local address 127.0.97.62:33533)
0926 03:32:06.879595 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:06.879603 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:06.879898 (+ 295us) server_negotiation.cc:407] Connection header received
0926 03:32:06.880130 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:06.880135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:06.880198 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:06.880313 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:06.881689 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:06.882392 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:06.883666 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:06.884096 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:06.884249 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:06.889927 (+ 5678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37066: BlockingRecv error: recv got EOF from 127.0.0.1:37066 (error 108)
Metrics: {"server-negotiator.queue_time_us":207}
Sep 26 03:32:06 dist-test-slave-dt7p krb5kdc[7920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393526, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:32:06.907234 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:06.893316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37082 (local address 127.0.97.62:33533)
0926 03:32:06.893553 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:06.893559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:06.893735 (+ 176us) server_negotiation.cc:407] Connection header received
0926 03:32:06.893863 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:06.893866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:06.893939 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:06.894023 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:06.895340 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:06.896166 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:06.897586 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:06.898002 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:06.898100 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:06.902046 (+ 3946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:06.902051 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:06.902054 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:06.902088 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:06.904531 (+ 2443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:06.905302 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:06.905309 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:06.905312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:06.905438 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:06.905809 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:06.905815 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:06.905818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:06.906106 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:06.906388 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:06.906682 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 03:32:06.906922 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 03:32:06.911432 7965 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:32:06.918687 7965 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:32:06.954049 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:06.930647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54358 (local address 127.0.97.3:43645)
0926 03:32:06.944672 (+ 14025us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:06.944678 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:06.944702 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:32:06.944771 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:06.944776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:06.945093 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:06.945279 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:32:06.946258 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:06.947868 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:06.951066 (+ 3198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:06.952481 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:06.952638 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:06.952862 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:06.953554 (+ 692us) server_negotiation.cc:299] Negotiation successful
0926 03:32:06.953720 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13826,"thread_start_us":13643,"threads_started":1}
I20260926 03:32:06.954116 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:06.930479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43645 (local address 127.0.0.1:54358)
0926 03:32:06.943060 (+ 12581us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:06.943110 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:06.943284 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:06.945316 (+ 2032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:06.945325 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:06.945356 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:32:06.945988 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:06.946009 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:06.947998 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:06.948003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:06.950877 (+ 2874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:06.950894 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:06.953042 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:06.953047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:06.953251 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:06.953347 (+ 96us) client_negotiation.cc:769] Sending connection context
0926 03:32:06.953488 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 03:32:06.953700 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12272,"mutex_wait_us":320,"thread_start_us":12886,"threads_started":1}
I20260926 03:32:06.954102 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:06.944632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54662 (local address 127.0.97.2:35309)
0926 03:32:06.945211 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:06.945220 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:06.946305 (+ 1085us) server_negotiation.cc:407] Connection header received
0926 03:32:06.946584 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:06.946594 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:06.946804 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:06.946978 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:32:06.947558 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:06.948950 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:06.951139 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:06.952488 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:06.952630 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:06.952821 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:06.953630 (+ 809us) server_negotiation.cc:299] Negotiation successful
0926 03:32:06.953794 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":293,"threads_started":1}
I20260926 03:32:06.954149 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:06.931224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60342 (local address 127.0.97.1:46445)
0926 03:32:06.943500 (+ 12276us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:06.943513 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:06.944325 (+ 812us) server_negotiation.cc:407] Connection header received
0926 03:32:06.944623 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:06.944632 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:06.944942 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:06.945137 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:32:06.946313 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:06.947821 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:06.951045 (+ 3224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:06.952477 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:06.952623 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:06.952829 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:06.953730 (+ 901us) server_negotiation.cc:299] Negotiation successful
0926 03:32:06.953865 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12051,"thread_start_us":12280,"threads_started":1}
I20260926 03:32:06.954833 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:06.930620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46445 (local address 127.0.0.1:60342)
0926 03:32:06.944186 (+ 13566us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:06.944213 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:06.944432 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:06.945397 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:06.945405 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:06.945446 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:32:06.946106 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:06.946116 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:06.947969 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:06.947984 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:06.950876 (+ 2892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:06.950894 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:06.953021 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:06.953027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:06.953267 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:06.953356 (+ 89us) client_negotiation.cc:769] Sending connection context
0926 03:32:06.953469 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 03:32:06.954442 (+ 973us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13441,"mutex_wait_us":980,"thread_start_us":13287,"threads_started":1}
I20260926 03:32:06.954900 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:06.945208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35309 (local address 127.0.0.1:54662)
0926 03:32:06.946182 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:06.946199 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:06.946406 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:06.947002 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:06.947006 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:06.947029 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:32:06.947377 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:06.947388 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:06.949087 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:06.949093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:06.950938 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:06.950949 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:06.952968 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:06.952975 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:06.953272 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:06.953348 (+ 76us) client_negotiation.cc:769] Sending connection context
0926 03:32:06.953478 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 03:32:06.954615 (+ 1137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"mutex_wait_us":1030,"thread_start_us":590,"threads_started":1}
I20260926 03:32:06.958025 8241 tablet_service.cc:1467] Processing CreateTablet for tablet e4d1dd6c919c421ab2d9fcd4d165517d (DEFAULT_TABLE table=test-table [id=afb20b25ac3a4144b57659f99183836c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:32:06.958081 8383 tablet_service.cc:1467] Processing CreateTablet for tablet e4d1dd6c919c421ab2d9fcd4d165517d (DEFAULT_TABLE table=test-table [id=afb20b25ac3a4144b57659f99183836c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:32:06.958619 8102 tablet_service.cc:1467] Processing CreateTablet for tablet e4d1dd6c919c421ab2d9fcd4d165517d (DEFAULT_TABLE table=test-table [id=afb20b25ac3a4144b57659f99183836c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:32:06.959399 8241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4d1dd6c919c421ab2d9fcd4d165517d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:32:06.959438 8383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4d1dd6c919c421ab2d9fcd4d165517d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:32:06.959769 8102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4d1dd6c919c421ab2d9fcd4d165517d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:32:06.969799 8474 tablet_bootstrap.cc:492] T e4d1dd6c919c421ab2d9fcd4d165517d P 7b178dcd80ae46949a1793f90f4a5cf9: Bootstrap starting.
I20260926 03:32:06.972941 8474 tablet_bootstrap.cc:654] T e4d1dd6c919c421ab2d9fcd4d165517d P 7b178dcd80ae46949a1793f90f4a5cf9: Neither blocks nor log segments found. Creating new log.
I20260926 03:32:06.973531 8475 tablet_bootstrap.cc:492] T e4d1dd6c919c421ab2d9fcd4d165517d P b2b9ca83586547e18d54d7eb8eed7b76: Bootstrap starting.
I20260926 03:32:06.975714 8474 log.cc:824] T e4d1dd6c919c421ab2d9fcd4d165517d P 7b178dcd80ae46949a1793f90f4a5cf9: Log is configured to *not* fsync() on all Append() calls
I20260926 03:32:06.977058 8475 tablet_bootstrap.cc:654] T e4d1dd6c919c421ab2d9fcd4d165517d P b2b9ca83586547e18d54d7eb8eed7b76: Neither blocks nor log segments found. Creating new log.
I20260926 03:32:06.977123 8476 tablet_bootstrap.cc:492] T e4d1dd6c919c421ab2d9fcd4d165517d P 261504269b4c4c7e9d96e8fd93cff7dc: Bootstrap starting.
I20260926 03:32:06.978281 8475 log.cc:824] T e4d1dd6c919c421ab2d9fcd4d165517d P b2b9ca83586547e18d54d7eb8eed7b76: Log is configured to *not* fsync() on all Append() calls
I20260926 03:32:06.980638 8474 tablet_bootstrap.cc:492] T e4d1dd6c919c421ab2d9fcd4d165517d P 7b178dcd80ae46949a1793f90f4a5cf9: No bootstrap required, opened a new log
I20260926 03:32:06.980907 8474 ts_tablet_manager.cc:1397] T e4d1dd6c919c421ab2d9fcd4d165517d P 7b178dcd80ae46949a1793f90f4a5cf9: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260926 03:32:06.981001 8475 tablet_bootstrap.cc:492] T e4d1dd6c919c421ab2d9fcd4d165517d P b2b9ca83586547e18d54d7eb8eed7b76: No bootstrap required, opened a new log
I20260926 03:32:06.981241 8475 ts_tablet_manager.cc:1397] T e4d1dd6c919c421ab2d9fcd4d165517d P b2b9ca83586547e18d54d7eb8eed7b76: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 03:32:06.982197 8476 tablet_bootstrap.cc:654] T e4d1dd6c919c421ab2d9fcd4d165517d P 261504269b4c4c7e9d96e8fd93cff7dc: Neither blocks nor log segments found. Creating new log.
I20260926 03:32:06.983129 8476 log.cc:824] T e4d1dd6c919c421ab2d9fcd4d165517d P 261504269b4c4c7e9d96e8fd93cff7dc: Log is configured to *not* fsync() on all Append() calls
I20260926 03:32:06.986135 8476 tablet_bootstrap.cc:492] T e4d1dd6c919c421ab2d9fcd4d165517d P 261504269b4c4c7e9d96e8fd93cff7dc: No bootstrap required, opened a new log
I20260926 03:32:06.986384 8476 ts_tablet_manager.cc:1397] T e4d1dd6c919c421ab2d9fcd4d165517d P 261504269b4c4c7e9d96e8fd93cff7dc: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260926 03:32:06.991699 8475 raft_consensus.cc:348] T e4d1dd6c919c421ab2d9fcd4d165517d P b2b9ca83586547e18d54d7eb8eed7b76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } } peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } }
I20260926 03:32:06.992170 8475 raft_consensus.cc:374] T e4d1dd6c919c421ab2d9fcd4d165517d P b2b9ca83586547e18d54d7eb8eed7b76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:32:06.992277 8475 raft_consensus.cc:729] T e4d1dd6c919c421ab2d9fcd4d165517d P b2b9ca83586547e18d54d7eb8eed7b76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2b9ca83586547e18d54d7eb8eed7b76, State: Initialized, Role: FOLLOWER
I20260926 03:32:06.992182 8474 raft_consensus.cc:348] T e4d1dd6c919c421ab2d9fcd4d165517d P 7b178dcd80ae46949a1793f90f4a5cf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } } peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } }
I20260926 03:32:06.992579 8474 raft_consensus.cc:374] T e4d1dd6c919c421ab2d9fcd4d165517d P 7b178dcd80ae46949a1793f90f4a5cf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:32:06.992690 8474 raft_consensus.cc:729] T e4d1dd6c919c421ab2d9fcd4d165517d P 7b178dcd80ae46949a1793f90f4a5cf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b178dcd80ae46949a1793f90f4a5cf9, State: Initialized, Role: FOLLOWER
I20260926 03:32:06.993072 8475 consensus_queue.cc:260] T e4d1dd6c919c421ab2d9fcd4d165517d P b2b9ca83586547e18d54d7eb8eed7b76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } } peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } }
I20260926 03:32:06.993201 8474 consensus_queue.cc:260] T e4d1dd6c919c421ab2d9fcd4d165517d P 7b178dcd80ae46949a1793f90f4a5cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } } peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } }
I20260926 03:32:06.995078 8475 ts_tablet_manager.cc:1428] T e4d1dd6c919c421ab2d9fcd4d165517d P b2b9ca83586547e18d54d7eb8eed7b76: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s
I20260926 03:32:06.996011 8311 heartbeater.cc:503] Master 127.0.97.62:33533 was elected leader, sending a full tablet report...
I20260926 03:32:06.996407 8474 ts_tablet_manager.cc:1428] T e4d1dd6c919c421ab2d9fcd4d165517d P 7b178dcd80ae46949a1793f90f4a5cf9: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
I20260926 03:32:06.998529 8476 raft_consensus.cc:348] T e4d1dd6c919c421ab2d9fcd4d165517d P 261504269b4c4c7e9d96e8fd93cff7dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } } peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } }
I20260926 03:32:06.998939 8476 raft_consensus.cc:374] T e4d1dd6c919c421ab2d9fcd4d165517d P 261504269b4c4c7e9d96e8fd93cff7dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:32:06.999042 8476 raft_consensus.cc:729] T e4d1dd6c919c421ab2d9fcd4d165517d P 261504269b4c4c7e9d96e8fd93cff7dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 261504269b4c4c7e9d96e8fd93cff7dc, State: Initialized, Role: FOLLOWER
I20260926 03:32:06.999614 8476 consensus_queue.cc:260] T e4d1dd6c919c421ab2d9fcd4d165517d P 261504269b4c4c7e9d96e8fd93cff7dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } } peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } }
I20260926 03:32:07.000658 8450 heartbeater.cc:503] Master 127.0.97.62:33533 was elected leader, sending a full tablet report...
I20260926 03:32:07.002158 8476 ts_tablet_manager.cc:1428] T e4d1dd6c919c421ab2d9fcd4d165517d P 261504269b4c4c7e9d96e8fd93cff7dc: Time spent starting tablet: real 0.016s user 0.011s sys 0.000s
W20260926 03:32:07.030241 8173 tablet.cc:2369] T e4d1dd6c919c421ab2d9fcd4d165517d P b2b9ca83586547e18d54d7eb8eed7b76: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:32:07.056490 8312 tablet.cc:2369] T e4d1dd6c919c421ab2d9fcd4d165517d P 7b178dcd80ae46949a1793f90f4a5cf9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:32:07.086058 8451 tablet.cc:2369] T e4d1dd6c919c421ab2d9fcd4d165517d P 261504269b4c4c7e9d96e8fd93cff7dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:32:07.265698 8482 raft_consensus.cc:482] T e4d1dd6c919c421ab2d9fcd4d165517d P 261504269b4c4c7e9d96e8fd93cff7dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:32:07.266000 8482 raft_consensus.cc:504] T e4d1dd6c919c421ab2d9fcd4d165517d P 261504269b4c4c7e9d96e8fd93cff7dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } } peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } }
I20260926 03:32:07.267597 8482 leader_election.cc:288] T e4d1dd6c919c421ab2d9fcd4d165517d P 261504269b4c4c7e9d96e8fd93cff7dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2b9ca83586547e18d54d7eb8eed7b76 (127.0.97.1:46445), 7b178dcd80ae46949a1793f90f4a5cf9 (127.0.97.2:35309)
I20260926 03:32:07.276324 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:07.268028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49513 (local address 127.0.97.1:46445)
0926 03:32:07.268214 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:07.268218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:07.268769 (+ 551us) server_negotiation.cc:407] Connection header received
0926 03:32:07.268908 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:07.268912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:07.269039 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:07.269124 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:32:07.269843 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:07.270856 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:07.273911 (+ 3055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:07.275181 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:07.275298 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:07.275390 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:07.275938 (+ 548us) server_negotiation.cc:299] Negotiation successful
0926 03:32:07.276087 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 03:32:07.276327 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:07.268044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56139 (local address 127.0.97.2:35309)
0926 03:32:07.268261 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:07.268269 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:07.268534 (+ 265us) server_negotiation.cc:407] Connection header received
0926 03:32:07.268822 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:07.268827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:07.268921 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:07.269025 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:32:07.269696 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:07.270817 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:07.273910 (+ 3093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:07.275141 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:07.275287 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:07.275390 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:07.275937 (+ 547us) server_negotiation.cc:299] Negotiation successful
0926 03:32:07.276062 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 03:32:07.276597 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:07.268068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46445 (local address 127.0.97.3:49513)
0926 03:32:07.268664 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:07.268681 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:07.268790 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:07.269168 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:07.269173 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:07.269191 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:32:07.269690 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:07.269704 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:07.271005 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:07.271011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:07.273724 (+ 2713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:07.273739 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:07.275408 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:07.275413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:07.275701 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:07.275798 (+ 97us) client_negotiation.cc:769] Sending connection context
0926 03:32:07.275902 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 03:32:07.276085 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":279,"thread_start_us":316,"threads_started":1}
I20260926 03:32:07.277210 8339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:07.267930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35309 (local address 127.0.97.3:56139)
0926 03:32:07.268404 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:07.268428 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:07.268638 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:07.269127 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:07.269134 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:07.269169 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:32:07.269534 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:07.269555 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:07.271005 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:07.271011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:07.273725 (+ 2714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:07.273739 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:07.275368 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:07.275377 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:07.275723 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:07.275806 (+ 83us) client_negotiation.cc:769] Sending connection context
0926 03:32:07.275902 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 03:32:07.276085 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":356}
I20260926 03:32:07.277673 8127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4d1dd6c919c421ab2d9fcd4d165517d" candidate_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" is_pre_election: true
I20260926 03:32:07.278167 8127 raft_consensus.cc:2454] T e4d1dd6c919c421ab2d9fcd4d165517d P b2b9ca83586547e18d54d7eb8eed7b76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 261504269b4c4c7e9d96e8fd93cff7dc in term 0.
I20260926 03:32:07.278117 8261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4d1dd6c919c421ab2d9fcd4d165517d" candidate_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" is_pre_election: true
I20260926 03:32:07.278606 8261 raft_consensus.cc:2454] T e4d1dd6c919c421ab2d9fcd4d165517d P 7b178dcd80ae46949a1793f90f4a5cf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 261504269b4c4c7e9d96e8fd93cff7dc in term 0.
I20260926 03:32:07.278867 8334 leader_election.cc:302] T e4d1dd6c919c421ab2d9fcd4d165517d P 261504269b4c4c7e9d96e8fd93cff7dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 261504269b4c4c7e9d96e8fd93cff7dc, b2b9ca83586547e18d54d7eb8eed7b76; no voters:
I20260926 03:32:07.279253 8482 raft_consensus.cc:2781] T e4d1dd6c919c421ab2d9fcd4d165517d P 261504269b4c4c7e9d96e8fd93cff7dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:32:07.279397 8482 raft_consensus.cc:482] T e4d1dd6c919c421ab2d9fcd4d165517d P 261504269b4c4c7e9d96e8fd93cff7dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:32:07.279507 8482 raft_consensus.cc:3037] T e4d1dd6c919c421ab2d9fcd4d165517d P 261504269b4c4c7e9d96e8fd93cff7dc [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:32:07.281797 8482 raft_consensus.cc:504] T e4d1dd6c919c421ab2d9fcd4d165517d P 261504269b4c4c7e9d96e8fd93cff7dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } } peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } }
I20260926 03:32:07.282442 8482 leader_election.cc:288] T e4d1dd6c919c421ab2d9fcd4d165517d P 261504269b4c4c7e9d96e8fd93cff7dc [CANDIDATE]: Term 1 election: Requested vote from peers b2b9ca83586547e18d54d7eb8eed7b76 (127.0.97.1:46445), 7b178dcd80ae46949a1793f90f4a5cf9 (127.0.97.2:35309)
I20260926 03:32:07.282829 8127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4d1dd6c919c421ab2d9fcd4d165517d" candidate_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2b9ca83586547e18d54d7eb8eed7b76"
I20260926 03:32:07.282895 8261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4d1dd6c919c421ab2d9fcd4d165517d" candidate_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b178dcd80ae46949a1793f90f4a5cf9"
I20260926 03:32:07.283025 8127 raft_consensus.cc:3037] T e4d1dd6c919c421ab2d9fcd4d165517d P b2b9ca83586547e18d54d7eb8eed7b76 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:32:07.283139 8261 raft_consensus.cc:3037] T e4d1dd6c919c421ab2d9fcd4d165517d P 7b178dcd80ae46949a1793f90f4a5cf9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:32:07.285337 8127 raft_consensus.cc:2454] T e4d1dd6c919c421ab2d9fcd4d165517d P b2b9ca83586547e18d54d7eb8eed7b76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 261504269b4c4c7e9d96e8fd93cff7dc in term 1.
I20260926 03:32:07.285732 8334 leader_election.cc:302] T e4d1dd6c919c421ab2d9fcd4d165517d P 261504269b4c4c7e9d96e8fd93cff7dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 261504269b4c4c7e9d96e8fd93cff7dc, b2b9ca83586547e18d54d7eb8eed7b76; no voters:
I20260926 03:32:07.285802 8261 raft_consensus.cc:2454] T e4d1dd6c919c421ab2d9fcd4d165517d P 7b178dcd80ae46949a1793f90f4a5cf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 261504269b4c4c7e9d96e8fd93cff7dc in term 1.
I20260926 03:32:07.286059 8482 raft_consensus.cc:2781] T e4d1dd6c919c421ab2d9fcd4d165517d P 261504269b4c4c7e9d96e8fd93cff7dc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:32:07.286510 8482 raft_consensus.cc:686] T e4d1dd6c919c421ab2d9fcd4d165517d P 261504269b4c4c7e9d96e8fd93cff7dc [term 1 LEADER]: Becoming Leader. State: Replica: 261504269b4c4c7e9d96e8fd93cff7dc, State: Running, Role: LEADER
I20260926 03:32:07.286919 8482 consensus_queue.cc:237] T e4d1dd6c919c421ab2d9fcd4d165517d P 261504269b4c4c7e9d96e8fd93cff7dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } } peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } }
I20260926 03:32:07.291715 7964 catalog_manager.cc:5495] T e4d1dd6c919c421ab2d9fcd4d165517d P 261504269b4c4c7e9d96e8fd93cff7dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 261504269b4c4c7e9d96e8fd93cff7dc (127.0.97.3). New cstate: current_term: 1 leader_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } health_report { overall_health: UNKNOWN } } }
I20260926 03:32:07.311825 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:07.306444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54366 (local address 127.0.97.3:43645)
0926 03:32:07.306649 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:07.306655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:07.306684 (+ 29us) server_negotiation.cc:407] Connection header received
0926 03:32:07.306816 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:07.306821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:07.306907 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:07.307078 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:32:07.307430 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:07.308282 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:07.309835 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:07.310240 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:07.310374 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:07.310886 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:32:07.311348 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:32:07.311460 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:07.311565 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 03:32:07.311633 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 03:32:07.319197 8261 raft_consensus.cc:1261] T e4d1dd6c919c421ab2d9fcd4d165517d P 7b178dcd80ae46949a1793f90f4a5cf9 [term 1 FOLLOWER]: Refusing update from remote peer 261504269b4c4c7e9d96e8fd93cff7dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:32:07.319237 8127 raft_consensus.cc:1261] T e4d1dd6c919c421ab2d9fcd4d165517d P b2b9ca83586547e18d54d7eb8eed7b76 [term 1 FOLLOWER]: Refusing update from remote peer 261504269b4c4c7e9d96e8fd93cff7dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:32:07.320114 8484 consensus_queue.cc:1035] T e4d1dd6c919c421ab2d9fcd4d165517d P 261504269b4c4c7e9d96e8fd93cff7dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:32:07.320474 8482 consensus_queue.cc:1035] T e4d1dd6c919c421ab2d9fcd4d165517d P 261504269b4c4c7e9d96e8fd93cff7dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:32:07.338025 8487 mvcc.cc:205] Tried to move back new op lower bound from 7333451887890927616 to 7333451887770923008. Current Snapshot: MvccSnapshot[applied={T|T < 7333451887890927616}]
I20260926 03:32:07.339466 8489 mvcc.cc:205] Tried to move back new op lower bound from 7333451887890927616 to 7333451887770923008. Current Snapshot: MvccSnapshot[applied={T|T < 7333451887890927616}]
I20260926 03:32:07.351786 7964 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:32:07.352089 7964 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:32:07.355610 7964 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dbe96762262e460388583b79167d2c1d: Sending DeleteTablet for 3 replicas of tablet e4d1dd6c919c421ab2d9fcd4d165517d
I20260926 03:32:07.356736 8383 tablet_service.cc:1514] Processing DeleteTablet for tablet e4d1dd6c919c421ab2d9fcd4d165517d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:32:07 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54358
I20260926 03:32:07.356750 8102 tablet_service.cc:1514] Processing DeleteTablet for tablet e4d1dd6c919c421ab2d9fcd4d165517d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:32:07 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60342
I20260926 03:32:07.357002 8241 tablet_service.cc:1514] Processing DeleteTablet for tablet e4d1dd6c919c421ab2d9fcd4d165517d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:32:07 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54662
I20260926 03:32:07.359422 7964 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:32:07.360821 7964 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:32:07.372555 8499 tablet_replica.cc:330] T e4d1dd6c919c421ab2d9fcd4d165517d P 7b178dcd80ae46949a1793f90f4a5cf9: stopping tablet replica
I20260926 03:32:07.373287 8499 raft_consensus.cc:2229] T e4d1dd6c919c421ab2d9fcd4d165517d P 7b178dcd80ae46949a1793f90f4a5cf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:32:07.373560 8379 tablet_service.cc:1467] Processing CreateTablet for tablet 6bdd021f80f448bda941975be737c605 (DEFAULT_TABLE table=test-table [id=aec1060acd2941ef81353000f7ba5a0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:32:07.373989 8499 raft_consensus.cc:2258] T e4d1dd6c919c421ab2d9fcd4d165517d P 7b178dcd80ae46949a1793f90f4a5cf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:32:07.374133 8379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bdd021f80f448bda941975be737c605. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:32:07.376175 8499 ts_tablet_manager.cc:1905] T e4d1dd6c919c421ab2d9fcd4d165517d P 7b178dcd80ae46949a1793f90f4a5cf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:32:07.377494 8101 tablet_service.cc:1467] Processing CreateTablet for tablet 6bdd021f80f448bda941975be737c605 (DEFAULT_TABLE table=test-table [id=aec1060acd2941ef81353000f7ba5a0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:32:07.377974 8101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bdd021f80f448bda941975be737c605. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:32:07.382035 8241 tablet_service.cc:1467] Processing CreateTablet for tablet 6bdd021f80f448bda941975be737c605 (DEFAULT_TABLE table=test-table [id=aec1060acd2941ef81353000f7ba5a0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:32:07.382253 8476 tablet_bootstrap.cc:492] T 6bdd021f80f448bda941975be737c605 P 261504269b4c4c7e9d96e8fd93cff7dc: Bootstrap starting.
I20260926 03:32:07.382488 8241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bdd021f80f448bda941975be737c605. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:32:07.384485 8476 tablet_bootstrap.cc:654] T 6bdd021f80f448bda941975be737c605 P 261504269b4c4c7e9d96e8fd93cff7dc: Neither blocks nor log segments found. Creating new log.
I20260926 03:32:07.401389 8475 tablet_bootstrap.cc:492] T 6bdd021f80f448bda941975be737c605 P b2b9ca83586547e18d54d7eb8eed7b76: Bootstrap starting.
I20260926 03:32:07.403407 8475 tablet_bootstrap.cc:654] T 6bdd021f80f448bda941975be737c605 P b2b9ca83586547e18d54d7eb8eed7b76: Neither blocks nor log segments found. Creating new log.
I20260926 03:32:07.403585 8476 tablet_bootstrap.cc:492] T 6bdd021f80f448bda941975be737c605 P 261504269b4c4c7e9d96e8fd93cff7dc: No bootstrap required, opened a new log
I20260926 03:32:07.403748 8476 ts_tablet_manager.cc:1397] T 6bdd021f80f448bda941975be737c605 P 261504269b4c4c7e9d96e8fd93cff7dc: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.008s
I20260926 03:32:07.404661 8476 raft_consensus.cc:348] T 6bdd021f80f448bda941975be737c605 P 261504269b4c4c7e9d96e8fd93cff7dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } }
I20260926 03:32:07.405002 8476 raft_consensus.cc:374] T 6bdd021f80f448bda941975be737c605 P 261504269b4c4c7e9d96e8fd93cff7dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:32:07.405120 8476 raft_consensus.cc:729] T 6bdd021f80f448bda941975be737c605 P 261504269b4c4c7e9d96e8fd93cff7dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 261504269b4c4c7e9d96e8fd93cff7dc, State: Initialized, Role: FOLLOWER
I20260926 03:32:07.405526 8476 consensus_queue.cc:260] T 6bdd021f80f448bda941975be737c605 P 261504269b4c4c7e9d96e8fd93cff7dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } }
I20260926 03:32:07.406303 8476 ts_tablet_manager.cc:1428] T 6bdd021f80f448bda941975be737c605 P 261504269b4c4c7e9d96e8fd93cff7dc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 03:32:07.410068 8497 tablet_replica.cc:330] T e4d1dd6c919c421ab2d9fcd4d165517d P 261504269b4c4c7e9d96e8fd93cff7dc: stopping tablet replica
I20260926 03:32:07.410614 8498 tablet_replica.cc:330] T e4d1dd6c919c421ab2d9fcd4d165517d P b2b9ca83586547e18d54d7eb8eed7b76: stopping tablet replica
I20260926 03:32:07.411036 8498 raft_consensus.cc:2229] T e4d1dd6c919c421ab2d9fcd4d165517d P b2b9ca83586547e18d54d7eb8eed7b76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:32:07.411623 8498 raft_consensus.cc:2258] T e4d1dd6c919c421ab2d9fcd4d165517d P b2b9ca83586547e18d54d7eb8eed7b76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:32:07.412370 8474 tablet_bootstrap.cc:492] T 6bdd021f80f448bda941975be737c605 P 7b178dcd80ae46949a1793f90f4a5cf9: Bootstrap starting.
I20260926 03:32:07.412947 8497 raft_consensus.cc:2229] T e4d1dd6c919c421ab2d9fcd4d165517d P 261504269b4c4c7e9d96e8fd93cff7dc [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:32:07.413591 8497 raft_consensus.cc:2258] T e4d1dd6c919c421ab2d9fcd4d165517d P 261504269b4c4c7e9d96e8fd93cff7dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:32:07.414178 8474 tablet_bootstrap.cc:654] T 6bdd021f80f448bda941975be737c605 P 7b178dcd80ae46949a1793f90f4a5cf9: Neither blocks nor log segments found. Creating new log.
I20260926 03:32:07.415659 8497 ts_tablet_manager.cc:1905] T e4d1dd6c919c421ab2d9fcd4d165517d P 261504269b4c4c7e9d96e8fd93cff7dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:32:07.418041 8474 tablet_bootstrap.cc:492] T 6bdd021f80f448bda941975be737c605 P 7b178dcd80ae46949a1793f90f4a5cf9: No bootstrap required, opened a new log
I20260926 03:32:07.418177 8474 ts_tablet_manager.cc:1397] T 6bdd021f80f448bda941975be737c605 P 7b178dcd80ae46949a1793f90f4a5cf9: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20260926 03:32:07.418239 8498 ts_tablet_manager.cc:1905] T e4d1dd6c919c421ab2d9fcd4d165517d P b2b9ca83586547e18d54d7eb8eed7b76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:32:07.417197 8499 ts_tablet_manager.cc:1918] T e4d1dd6c919c421ab2d9fcd4d165517d P 7b178dcd80ae46949a1793f90f4a5cf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:32:07.418625 8499 log.cc:1196] T e4d1dd6c919c421ab2d9fcd4d165517d P 7b178dcd80ae46949a1793f90f4a5cf9: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-1/wal/wals/e4d1dd6c919c421ab2d9fcd4d165517d
I20260926 03:32:07.419057 8474 raft_consensus.cc:348] T 6bdd021f80f448bda941975be737c605 P 7b178dcd80ae46949a1793f90f4a5cf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } }
I20260926 03:32:07.419277 8474 raft_consensus.cc:374] T 6bdd021f80f448bda941975be737c605 P 7b178dcd80ae46949a1793f90f4a5cf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:32:07.419363 8474 raft_consensus.cc:729] T 6bdd021f80f448bda941975be737c605 P 7b178dcd80ae46949a1793f90f4a5cf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b178dcd80ae46949a1793f90f4a5cf9, State: Initialized, Role: FOLLOWER
I20260926 03:32:07.419394 8499 ts_tablet_manager.cc:1939] T e4d1dd6c919c421ab2d9fcd4d165517d P 7b178dcd80ae46949a1793f90f4a5cf9: Deleting consensus metadata
I20260926 03:32:07.419667 8474 consensus_queue.cc:260] T 6bdd021f80f448bda941975be737c605 P 7b178dcd80ae46949a1793f90f4a5cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } }
I20260926 03:32:07.420240 8474 ts_tablet_manager.cc:1428] T 6bdd021f80f448bda941975be737c605 P 7b178dcd80ae46949a1793f90f4a5cf9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260926 03:32:07.421952 7951 catalog_manager.cc:4841] TS 7b178dcd80ae46949a1793f90f4a5cf9 (127.0.97.2:35309): tablet e4d1dd6c919c421ab2d9fcd4d165517d (table test-table [id=afb20b25ac3a4144b57659f99183836c]) successfully deleted
I20260926 03:32:07.425701 8475 tablet_bootstrap.cc:492] T 6bdd021f80f448bda941975be737c605 P b2b9ca83586547e18d54d7eb8eed7b76: No bootstrap required, opened a new log
I20260926 03:32:07.425864 8475 ts_tablet_manager.cc:1397] T 6bdd021f80f448bda941975be737c605 P b2b9ca83586547e18d54d7eb8eed7b76: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.007s
I20260926 03:32:07.426374 8497 ts_tablet_manager.cc:1918] T e4d1dd6c919c421ab2d9fcd4d165517d P 261504269b4c4c7e9d96e8fd93cff7dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:32:07.426581 8497 log.cc:1196] T e4d1dd6c919c421ab2d9fcd4d165517d P 261504269b4c4c7e9d96e8fd93cff7dc: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-2/wal/wals/e4d1dd6c919c421ab2d9fcd4d165517d
I20260926 03:32:07.426705 8475 raft_consensus.cc:348] T 6bdd021f80f448bda941975be737c605 P b2b9ca83586547e18d54d7eb8eed7b76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } }
I20260926 03:32:07.427075 8497 ts_tablet_manager.cc:1939] T e4d1dd6c919c421ab2d9fcd4d165517d P 261504269b4c4c7e9d96e8fd93cff7dc: Deleting consensus metadata
I20260926 03:32:07.427057 8475 raft_consensus.cc:374] T 6bdd021f80f448bda941975be737c605 P b2b9ca83586547e18d54d7eb8eed7b76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:32:07.427178 8475 raft_consensus.cc:729] T 6bdd021f80f448bda941975be737c605 P b2b9ca83586547e18d54d7eb8eed7b76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2b9ca83586547e18d54d7eb8eed7b76, State: Initialized, Role: FOLLOWER
I20260926 03:32:07.427440 8475 consensus_queue.cc:260] T 6bdd021f80f448bda941975be737c605 P b2b9ca83586547e18d54d7eb8eed7b76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } }
I20260926 03:32:07.428105 8475 ts_tablet_manager.cc:1428] T 6bdd021f80f448bda941975be737c605 P b2b9ca83586547e18d54d7eb8eed7b76: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260926 03:32:07.428872 8498 ts_tablet_manager.cc:1918] T e4d1dd6c919c421ab2d9fcd4d165517d P b2b9ca83586547e18d54d7eb8eed7b76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:32:07.429016 8498 log.cc:1196] T e4d1dd6c919c421ab2d9fcd4d165517d P b2b9ca83586547e18d54d7eb8eed7b76: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-0/wal/wals/e4d1dd6c919c421ab2d9fcd4d165517d
I20260926 03:32:07.429548 8498 ts_tablet_manager.cc:1939] T e4d1dd6c919c421ab2d9fcd4d165517d P b2b9ca83586547e18d54d7eb8eed7b76: Deleting consensus metadata
I20260926 03:32:07.429642 7951 catalog_manager.cc:4841] TS 261504269b4c4c7e9d96e8fd93cff7dc (127.0.97.3:43645): tablet e4d1dd6c919c421ab2d9fcd4d165517d (table test-table [id=afb20b25ac3a4144b57659f99183836c]) successfully deleted
I20260926 03:32:07.431492 7952 catalog_manager.cc:4841] TS b2b9ca83586547e18d54d7eb8eed7b76 (127.0.97.1:46445): tablet e4d1dd6c919c421ab2d9fcd4d165517d (table test-table [id=afb20b25ac3a4144b57659f99183836c]) successfully deleted
I20260926 03:32:07.507241 8481 raft_consensus.cc:482] T 6bdd021f80f448bda941975be737c605 P 7b178dcd80ae46949a1793f90f4a5cf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:32:07.507529 8481 raft_consensus.cc:504] T 6bdd021f80f448bda941975be737c605 P 7b178dcd80ae46949a1793f90f4a5cf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } }
I20260926 03:32:07.509366 8481 leader_election.cc:288] T 6bdd021f80f448bda941975be737c605 P 7b178dcd80ae46949a1793f90f4a5cf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2b9ca83586547e18d54d7eb8eed7b76 (127.0.97.1:46445), 261504269b4c4c7e9d96e8fd93cff7dc (127.0.97.3:43645)
I20260926 03:32:07.516803 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:07.509926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47813 (local address 127.0.97.1:46445)
0926 03:32:07.510281 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:07.510287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:07.510861 (+ 574us) server_negotiation.cc:407] Connection header received
0926 03:32:07.511032 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:07.511037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:07.511119 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:07.511209 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:32:07.511947 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:07.513072 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:07.514869 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:07.515778 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:07.515902 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:07.516012 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:07.516356 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 03:32:07.516505 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247}
I20260926 03:32:07.517719 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:07.509632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46445 (local address 127.0.97.2:47813)
0926 03:32:07.510756 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:07.510773 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:07.510893 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:07.511248 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:07.511253 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:07.511280 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:32:07.511766 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:07.511778 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:07.513259 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:07.513263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:07.514724 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:07.514735 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:07.515982 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:07.515989 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:07.516162 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:07.516197 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 03:32:07.517049 (+ 852us) client_negotiation.cc:240] Negotiation successful
0926 03:32:07.517188 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":997,"mutex_wait_us":199,"thread_start_us":288,"threads_started":1}
I20260926 03:32:07.518296 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:07.509828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42481 (local address 127.0.97.3:43645)
0926 03:32:07.510143 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:07.510150 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:07.510546 (+ 396us) server_negotiation.cc:407] Connection header received
0926 03:32:07.510843 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:07.510851 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:07.510932 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:07.511031 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:32:07.511779 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:07.513062 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:07.515553 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:07.516731 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:07.516856 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:07.516940 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:07.517799 (+ 859us) server_negotiation.cc:299] Negotiation successful
0926 03:32:07.517984 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260926 03:32:07.518471 8127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bdd021f80f448bda941975be737c605" candidate_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" is_pre_election: true
I20260926 03:32:07.518721 8127 raft_consensus.cc:2454] T 6bdd021f80f448bda941975be737c605 P b2b9ca83586547e18d54d7eb8eed7b76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b178dcd80ae46949a1793f90f4a5cf9 in term 0.
I20260926 03:32:07.519251 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:07.509627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43645 (local address 127.0.97.2:42481)
0926 03:32:07.510417 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:07.510445 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:07.510670 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:07.511214 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:07.511218 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:07.511242 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:32:07.511610 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:07.511619 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:07.513197 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:07.513203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:07.515362 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:07.515380 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:07.517314 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:07.517323 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:07.517578 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:07.517618 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 03:32:07.518640 (+ 1022us) client_negotiation.cc:240] Negotiation successful
0926 03:32:07.518815 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":1,"thread_start_us":828,"threads_started":1}
I20260926 03:32:07.519330 8195 leader_election.cc:302] T 6bdd021f80f448bda941975be737c605 P 7b178dcd80ae46949a1793f90f4a5cf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b178dcd80ae46949a1793f90f4a5cf9, b2b9ca83586547e18d54d7eb8eed7b76; no voters:
I20260926 03:32:07.519770 8481 raft_consensus.cc:2781] T 6bdd021f80f448bda941975be737c605 P 7b178dcd80ae46949a1793f90f4a5cf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:32:07.519909 8481 raft_consensus.cc:482] T 6bdd021f80f448bda941975be737c605 P 7b178dcd80ae46949a1793f90f4a5cf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:32:07.520016 8481 raft_consensus.cc:3037] T 6bdd021f80f448bda941975be737c605 P 7b178dcd80ae46949a1793f90f4a5cf9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:32:07.520360 8405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bdd021f80f448bda941975be737c605" candidate_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" is_pre_election: true
I20260926 03:32:07.520821 8405 raft_consensus.cc:2454] T 6bdd021f80f448bda941975be737c605 P 261504269b4c4c7e9d96e8fd93cff7dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b178dcd80ae46949a1793f90f4a5cf9 in term 0.
I20260926 03:32:07.522701 8481 raft_consensus.cc:504] T 6bdd021f80f448bda941975be737c605 P 7b178dcd80ae46949a1793f90f4a5cf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } }
I20260926 03:32:07.523327 8481 leader_election.cc:288] T 6bdd021f80f448bda941975be737c605 P 7b178dcd80ae46949a1793f90f4a5cf9 [CANDIDATE]: Term 1 election: Requested vote from peers b2b9ca83586547e18d54d7eb8eed7b76 (127.0.97.1:46445), 261504269b4c4c7e9d96e8fd93cff7dc (127.0.97.3:43645)
I20260926 03:32:07.523834 8127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bdd021f80f448bda941975be737c605" candidate_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2b9ca83586547e18d54d7eb8eed7b76"
I20260926 03:32:07.523830 8405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bdd021f80f448bda941975be737c605" candidate_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "261504269b4c4c7e9d96e8fd93cff7dc"
I20260926 03:32:07.524040 8127 raft_consensus.cc:3037] T 6bdd021f80f448bda941975be737c605 P b2b9ca83586547e18d54d7eb8eed7b76 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:32:07.524047 8405 raft_consensus.cc:3037] T 6bdd021f80f448bda941975be737c605 P 261504269b4c4c7e9d96e8fd93cff7dc [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:32:07.527266 8127 raft_consensus.cc:2454] T 6bdd021f80f448bda941975be737c605 P b2b9ca83586547e18d54d7eb8eed7b76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b178dcd80ae46949a1793f90f4a5cf9 in term 1.
I20260926 03:32:07.527303 8405 raft_consensus.cc:2454] T 6bdd021f80f448bda941975be737c605 P 261504269b4c4c7e9d96e8fd93cff7dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b178dcd80ae46949a1793f90f4a5cf9 in term 1.
I20260926 03:32:07.527732 8195 leader_election.cc:302] T 6bdd021f80f448bda941975be737c605 P 7b178dcd80ae46949a1793f90f4a5cf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b178dcd80ae46949a1793f90f4a5cf9, b2b9ca83586547e18d54d7eb8eed7b76; no voters:
I20260926 03:32:07.528095 8481 raft_consensus.cc:2781] T 6bdd021f80f448bda941975be737c605 P 7b178dcd80ae46949a1793f90f4a5cf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:32:07.528584 8481 raft_consensus.cc:686] T 6bdd021f80f448bda941975be737c605 P 7b178dcd80ae46949a1793f90f4a5cf9 [term 1 LEADER]: Becoming Leader. State: Replica: 7b178dcd80ae46949a1793f90f4a5cf9, State: Running, Role: LEADER
I20260926 03:32:07.529019 8481 consensus_queue.cc:237] T 6bdd021f80f448bda941975be737c605 P 7b178dcd80ae46949a1793f90f4a5cf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } }
I20260926 03:32:07.532845 7963 catalog_manager.cc:5495] T 6bdd021f80f448bda941975be737c605 P 7b178dcd80ae46949a1793f90f4a5cf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b178dcd80ae46949a1793f90f4a5cf9 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } health_report { overall_health: UNKNOWN } } }
Sep 26 03:32:07 dist-test-slave-dt7p krb5kdc[7920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393525, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:32:07.604156 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:07.571044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33533 (local address 127.0.0.1:37086)
0926 03:32:07.571941 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:07.571957 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:07.572090 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:07.572443 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:07.572446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:07.573210 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:07.573423 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:07.573432 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:07.574889 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:07.574893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:07.575801 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:07.575811 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:07.576815 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:07.576820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:07.576944 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:07.595036 (+ 18092us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:07.595060 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:07.597762 (+ 2702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:32:07.601750 (+ 3988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:32:07.601762 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:32:07.601767 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:32:07.602188 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:32:07.602586 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:32:07.602589 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:32:07.602591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:32:07.602738 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:32:07.603365 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:32:07.603371 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:32:07.603510 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 03:32:07.603654 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 03:32:07.603883 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"thread_start_us":590,"threads_started":1}
I20260926 03:32:07.604480 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:07.571044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37086 (local address 127.0.97.62:33533)
0926 03:32:07.571753 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:07.571760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:07.572044 (+ 284us) server_negotiation.cc:407] Connection header received
0926 03:32:07.572203 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:07.572209 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:07.572287 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:07.572420 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:07.573590 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:07.574745 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:07.575934 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:07.576633 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:07.576750 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:07.598009 (+ 21259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:07.598018 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:07.598024 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:07.598066 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:07.601603 (+ 3537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:07.602319 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:07.602327 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:07.602331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:07.602477 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:07.602843 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:07.602851 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:07.602855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:07.603235 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:07.603417 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:07.603775 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 03:32:07.604036 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":526,"thread_start_us":324,"threads_started":1}
I20260926 03:32:07.609845 7963 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37086:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 03:32:07.610870 7963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:32:07.617832 8241 tablet_service.cc:1467] Processing CreateTablet for tablet d07df80224814d59a0d5f104f13f68e0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e66938f0f9d3432997335170fdfc08e2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:32:07.617832 8102 tablet_service.cc:1467] Processing CreateTablet for tablet d07df80224814d59a0d5f104f13f68e0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e66938f0f9d3432997335170fdfc08e2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:32:07.618413 8102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d07df80224814d59a0d5f104f13f68e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:32:07.618417 8241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d07df80224814d59a0d5f104f13f68e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:32:07.621587 8383 tablet_service.cc:1467] Processing CreateTablet for tablet d07df80224814d59a0d5f104f13f68e0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e66938f0f9d3432997335170fdfc08e2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:32:07.622159 8383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d07df80224814d59a0d5f104f13f68e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:32:07.625741 8474 tablet_bootstrap.cc:492] T d07df80224814d59a0d5f104f13f68e0 P 7b178dcd80ae46949a1793f90f4a5cf9: Bootstrap starting.
I20260926 03:32:07.626710 8475 tablet_bootstrap.cc:492] T d07df80224814d59a0d5f104f13f68e0 P b2b9ca83586547e18d54d7eb8eed7b76: Bootstrap starting.
I20260926 03:32:07.627432 8474 tablet_bootstrap.cc:654] T d07df80224814d59a0d5f104f13f68e0 P 7b178dcd80ae46949a1793f90f4a5cf9: Neither blocks nor log segments found. Creating new log.
I20260926 03:32:07.628443 8475 tablet_bootstrap.cc:654] T d07df80224814d59a0d5f104f13f68e0 P b2b9ca83586547e18d54d7eb8eed7b76: Neither blocks nor log segments found. Creating new log.
I20260926 03:32:07.629964 8476 tablet_bootstrap.cc:492] T d07df80224814d59a0d5f104f13f68e0 P 261504269b4c4c7e9d96e8fd93cff7dc: Bootstrap starting.
I20260926 03:32:07.631628 8476 tablet_bootstrap.cc:654] T d07df80224814d59a0d5f104f13f68e0 P 261504269b4c4c7e9d96e8fd93cff7dc: Neither blocks nor log segments found. Creating new log.
I20260926 03:32:07.638854 8474 tablet_bootstrap.cc:492] T d07df80224814d59a0d5f104f13f68e0 P 7b178dcd80ae46949a1793f90f4a5cf9: No bootstrap required, opened a new log
I20260926 03:32:07.639070 8474 ts_tablet_manager.cc:1397] T d07df80224814d59a0d5f104f13f68e0 P 7b178dcd80ae46949a1793f90f4a5cf9: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.010s
I20260926 03:32:07.640066 8474 raft_consensus.cc:348] T d07df80224814d59a0d5f104f13f68e0 P 7b178dcd80ae46949a1793f90f4a5cf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } }
I20260926 03:32:07.640395 8474 raft_consensus.cc:374] T d07df80224814d59a0d5f104f13f68e0 P 7b178dcd80ae46949a1793f90f4a5cf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:32:07.640503 8474 raft_consensus.cc:729] T d07df80224814d59a0d5f104f13f68e0 P 7b178dcd80ae46949a1793f90f4a5cf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b178dcd80ae46949a1793f90f4a5cf9, State: Initialized, Role: FOLLOWER
I20260926 03:32:07.641075 8474 consensus_queue.cc:260] T d07df80224814d59a0d5f104f13f68e0 P 7b178dcd80ae46949a1793f90f4a5cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } }
I20260926 03:32:07.641618 8481 tablet_replica.cc:439] T d07df80224814d59a0d5f104f13f68e0 P 7b178dcd80ae46949a1793f90f4a5cf9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } } }
I20260926 03:32:07.641902 8481 tablet_replica.cc:442] T d07df80224814d59a0d5f104f13f68e0 P 7b178dcd80ae46949a1793f90f4a5cf9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:32:07.642910 8475 tablet_bootstrap.cc:492] T d07df80224814d59a0d5f104f13f68e0 P b2b9ca83586547e18d54d7eb8eed7b76: No bootstrap required, opened a new log
I20260926 03:32:07.643087 8475 ts_tablet_manager.cc:1397] T d07df80224814d59a0d5f104f13f68e0 P b2b9ca83586547e18d54d7eb8eed7b76: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.004s
I20260926 03:32:07.643841 8475 raft_consensus.cc:348] T d07df80224814d59a0d5f104f13f68e0 P b2b9ca83586547e18d54d7eb8eed7b76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } }
I20260926 03:32:07.644107 8475 raft_consensus.cc:374] T d07df80224814d59a0d5f104f13f68e0 P b2b9ca83586547e18d54d7eb8eed7b76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:32:07.644197 8475 raft_consensus.cc:729] T d07df80224814d59a0d5f104f13f68e0 P b2b9ca83586547e18d54d7eb8eed7b76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2b9ca83586547e18d54d7eb8eed7b76, State: Initialized, Role: FOLLOWER
I20260926 03:32:07.644476 8475 consensus_queue.cc:260] T d07df80224814d59a0d5f104f13f68e0 P b2b9ca83586547e18d54d7eb8eed7b76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } }
I20260926 03:32:07.644763 8474 ts_tablet_manager.cc:1428] T d07df80224814d59a0d5f104f13f68e0 P 7b178dcd80ae46949a1793f90f4a5cf9: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20260926 03:32:07.644912 8476 tablet_bootstrap.cc:492] T d07df80224814d59a0d5f104f13f68e0 P 261504269b4c4c7e9d96e8fd93cff7dc: No bootstrap required, opened a new log
I20260926 03:32:07.645138 8476 ts_tablet_manager.cc:1397] T d07df80224814d59a0d5f104f13f68e0 P 261504269b4c4c7e9d96e8fd93cff7dc: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.009s
I20260926 03:32:07.645040 8480 tablet_replica.cc:439] T d07df80224814d59a0d5f104f13f68e0 P b2b9ca83586547e18d54d7eb8eed7b76: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } } }
I20260926 03:32:07.645377 8480 tablet_replica.cc:442] T d07df80224814d59a0d5f104f13f68e0 P b2b9ca83586547e18d54d7eb8eed7b76: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:32:07.646624 8475 ts_tablet_manager.cc:1428] T d07df80224814d59a0d5f104f13f68e0 P b2b9ca83586547e18d54d7eb8eed7b76: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 03:32:07.646152 8476 raft_consensus.cc:348] T d07df80224814d59a0d5f104f13f68e0 P 261504269b4c4c7e9d96e8fd93cff7dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } }
I20260926 03:32:07.647382 8476 raft_consensus.cc:374] T d07df80224814d59a0d5f104f13f68e0 P 261504269b4c4c7e9d96e8fd93cff7dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:32:07.647547 8476 raft_consensus.cc:729] T d07df80224814d59a0d5f104f13f68e0 P 261504269b4c4c7e9d96e8fd93cff7dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 261504269b4c4c7e9d96e8fd93cff7dc, State: Initialized, Role: FOLLOWER
I20260926 03:32:07.647852 8476 consensus_queue.cc:260] T d07df80224814d59a0d5f104f13f68e0 P 261504269b4c4c7e9d96e8fd93cff7dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } }
I20260926 03:32:07.648388 8492 tablet_replica.cc:439] T d07df80224814d59a0d5f104f13f68e0 P 261504269b4c4c7e9d96e8fd93cff7dc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } } }
I20260926 03:32:07.648849 8492 tablet_replica.cc:442] T d07df80224814d59a0d5f104f13f68e0 P 261504269b4c4c7e9d96e8fd93cff7dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:32:07.649530 8476 ts_tablet_manager.cc:1428] T d07df80224814d59a0d5f104f13f68e0 P 261504269b4c4c7e9d96e8fd93cff7dc: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260926 03:32:07.741469 8481 raft_consensus.cc:482] T d07df80224814d59a0d5f104f13f68e0 P 7b178dcd80ae46949a1793f90f4a5cf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:32:07.741724 8481 raft_consensus.cc:504] T d07df80224814d59a0d5f104f13f68e0 P 7b178dcd80ae46949a1793f90f4a5cf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } }
I20260926 03:32:07.742540 8481 leader_election.cc:288] T d07df80224814d59a0d5f104f13f68e0 P 7b178dcd80ae46949a1793f90f4a5cf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2b9ca83586547e18d54d7eb8eed7b76 (127.0.97.1:46445), 261504269b4c4c7e9d96e8fd93cff7dc (127.0.97.3:43645)
I20260926 03:32:07.743119 8127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d07df80224814d59a0d5f104f13f68e0" candidate_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" is_pre_election: true
I20260926 03:32:07.743129 8405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d07df80224814d59a0d5f104f13f68e0" candidate_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" is_pre_election: true
I20260926 03:32:07.743383 8127 raft_consensus.cc:2454] T d07df80224814d59a0d5f104f13f68e0 P b2b9ca83586547e18d54d7eb8eed7b76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b178dcd80ae46949a1793f90f4a5cf9 in term 0.
I20260926 03:32:07.743391 8405 raft_consensus.cc:2454] T d07df80224814d59a0d5f104f13f68e0 P 261504269b4c4c7e9d96e8fd93cff7dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b178dcd80ae46949a1793f90f4a5cf9 in term 0.
I20260926 03:32:07.743840 8195 leader_election.cc:302] T d07df80224814d59a0d5f104f13f68e0 P 7b178dcd80ae46949a1793f90f4a5cf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b178dcd80ae46949a1793f90f4a5cf9, b2b9ca83586547e18d54d7eb8eed7b76; no voters:
I20260926 03:32:07.744145 8481 raft_consensus.cc:2781] T d07df80224814d59a0d5f104f13f68e0 P 7b178dcd80ae46949a1793f90f4a5cf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:32:07.744248 8481 raft_consensus.cc:482] T d07df80224814d59a0d5f104f13f68e0 P 7b178dcd80ae46949a1793f90f4a5cf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:32:07.744362 8481 raft_consensus.cc:3037] T d07df80224814d59a0d5f104f13f68e0 P 7b178dcd80ae46949a1793f90f4a5cf9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:32:07.747462 8481 raft_consensus.cc:504] T d07df80224814d59a0d5f104f13f68e0 P 7b178dcd80ae46949a1793f90f4a5cf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } }
I20260926 03:32:07.748188 8481 leader_election.cc:288] T d07df80224814d59a0d5f104f13f68e0 P 7b178dcd80ae46949a1793f90f4a5cf9 [CANDIDATE]: Term 1 election: Requested vote from peers b2b9ca83586547e18d54d7eb8eed7b76 (127.0.97.1:46445), 261504269b4c4c7e9d96e8fd93cff7dc (127.0.97.3:43645)
I20260926 03:32:07.748621 8127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d07df80224814d59a0d5f104f13f68e0" candidate_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2b9ca83586547e18d54d7eb8eed7b76"
I20260926 03:32:07.748713 8405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d07df80224814d59a0d5f104f13f68e0" candidate_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "261504269b4c4c7e9d96e8fd93cff7dc"
I20260926 03:32:07.748868 8127 raft_consensus.cc:3037] T d07df80224814d59a0d5f104f13f68e0 P b2b9ca83586547e18d54d7eb8eed7b76 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:32:07.748967 8405 raft_consensus.cc:3037] T d07df80224814d59a0d5f104f13f68e0 P 261504269b4c4c7e9d96e8fd93cff7dc [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:32:07.751431 8405 raft_consensus.cc:2454] T d07df80224814d59a0d5f104f13f68e0 P 261504269b4c4c7e9d96e8fd93cff7dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b178dcd80ae46949a1793f90f4a5cf9 in term 1.
I20260926 03:32:07.751858 8194 leader_election.cc:302] T d07df80224814d59a0d5f104f13f68e0 P 7b178dcd80ae46949a1793f90f4a5cf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 261504269b4c4c7e9d96e8fd93cff7dc, 7b178dcd80ae46949a1793f90f4a5cf9; no voters:
I20260926 03:32:07.752097 8127 raft_consensus.cc:2454] T d07df80224814d59a0d5f104f13f68e0 P b2b9ca83586547e18d54d7eb8eed7b76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b178dcd80ae46949a1793f90f4a5cf9 in term 1.
I20260926 03:32:07.752136 8481 raft_consensus.cc:2781] T d07df80224814d59a0d5f104f13f68e0 P 7b178dcd80ae46949a1793f90f4a5cf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:32:07.752866 8481 raft_consensus.cc:686] T d07df80224814d59a0d5f104f13f68e0 P 7b178dcd80ae46949a1793f90f4a5cf9 [term 1 LEADER]: Becoming Leader. State: Replica: 7b178dcd80ae46949a1793f90f4a5cf9, State: Running, Role: LEADER
I20260926 03:32:07.753159 8481 consensus_queue.cc:237] T d07df80224814d59a0d5f104f13f68e0 P 7b178dcd80ae46949a1793f90f4a5cf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } }
I20260926 03:32:07.754860 8505 tablet_replica.cc:439] T d07df80224814d59a0d5f104f13f68e0 P 7b178dcd80ae46949a1793f90f4a5cf9: TxnStatusTablet state changed. Reason: New leader 7b178dcd80ae46949a1793f90f4a5cf9. Latest consensus state: current_term: 1 leader_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } } }
I20260926 03:32:07.755272 8505 tablet_replica.cc:442] T d07df80224814d59a0d5f104f13f68e0 P 7b178dcd80ae46949a1793f90f4a5cf9: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:32:07.756006 8518 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 03:32:07.756246 8518 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 03:32:07.756431 7963 catalog_manager.cc:5495] T d07df80224814d59a0d5f104f13f68e0 P 7b178dcd80ae46949a1793f90f4a5cf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b178dcd80ae46949a1793f90f4a5cf9 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } health_report { overall_health: UNKNOWN } } }
I20260926 03:32:07.815683 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:07.806420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35309 (local address 127.0.0.1:54676)
0926 03:32:07.806669 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:07.806689 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:07.806838 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:07.809371 (+ 2533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:07.809376 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:07.809390 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:32:07.809611 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:07.809622 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:07.811151 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:07.811156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:07.812766 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:07.812782 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:07.813763 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:07.813771 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:07.813965 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:07.814039 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:32:07.815097 (+ 1058us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:32:07.815116 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 03:32:07.815200 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 03:32:07.815369 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 03:32:07.815696 8521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:07.806839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54676 (local address 127.0.97.2:35309)
0926 03:32:07.808893 (+ 2054us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:07.808900 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:07.808920 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:32:07.809022 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:07.809030 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:07.809148 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:07.809314 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:32:07.809807 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:07.810987 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:07.812915 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:07.813602 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:07.813742 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:07.814250 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:32:07.814983 (+ 733us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:32:07.815119 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:07.815236 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 03:32:07.815317 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1884,"thread_start_us":1748,"threads_started":1}
I20260926 03:32:07.820498 8127 raft_consensus.cc:1261] T d07df80224814d59a0d5f104f13f68e0 P b2b9ca83586547e18d54d7eb8eed7b76 [term 1 FOLLOWER]: Refusing update from remote peer 7b178dcd80ae46949a1793f90f4a5cf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:32:07.820685 8405 raft_consensus.cc:1261] T d07df80224814d59a0d5f104f13f68e0 P 261504269b4c4c7e9d96e8fd93cff7dc [term 1 FOLLOWER]: Refusing update from remote peer 7b178dcd80ae46949a1793f90f4a5cf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:32:07.821431 8481 consensus_queue.cc:1035] T d07df80224814d59a0d5f104f13f68e0 P 7b178dcd80ae46949a1793f90f4a5cf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:32:07.822060 8481 consensus_queue.cc:1035] T d07df80224814d59a0d5f104f13f68e0 P 7b178dcd80ae46949a1793f90f4a5cf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:32:07.835871 8480 tablet_replica.cc:439] T d07df80224814d59a0d5f104f13f68e0 P b2b9ca83586547e18d54d7eb8eed7b76: TxnStatusTablet state changed. Reason: New leader 7b178dcd80ae46949a1793f90f4a5cf9. Latest consensus state: current_term: 1 leader_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } } }
I20260926 03:32:07.836184 8480 tablet_replica.cc:442] T d07df80224814d59a0d5f104f13f68e0 P b2b9ca83586547e18d54d7eb8eed7b76: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:32:07.837815 8481 tablet_replica.cc:439] T d07df80224814d59a0d5f104f13f68e0 P 7b178dcd80ae46949a1793f90f4a5cf9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } } }
I20260926 03:32:07.838109 8481 tablet_replica.cc:442] T d07df80224814d59a0d5f104f13f68e0 P 7b178dcd80ae46949a1793f90f4a5cf9: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:32:07.843720 8480 tablet_replica.cc:439] T d07df80224814d59a0d5f104f13f68e0 P b2b9ca83586547e18d54d7eb8eed7b76: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } } }
I20260926 03:32:07.843954 8480 tablet_replica.cc:442] T d07df80224814d59a0d5f104f13f68e0 P b2b9ca83586547e18d54d7eb8eed7b76: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:32:07.843961 8488 mvcc.cc:205] Tried to move back new op lower bound from 7333451889944637440 to 7333451889678876672. Current Snapshot: MvccSnapshot[applied={T|T < 7333451889944637440 or (T in {7333451889944637440})}]
I20260926 03:32:07.845427 8492 tablet_replica.cc:439] T d07df80224814d59a0d5f104f13f68e0 P 261504269b4c4c7e9d96e8fd93cff7dc: TxnStatusTablet state changed. Reason: New leader 7b178dcd80ae46949a1793f90f4a5cf9. Latest consensus state: current_term: 1 leader_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } } }
I20260926 03:32:07.845741 8492 tablet_replica.cc:442] T d07df80224814d59a0d5f104f13f68e0 P 261504269b4c4c7e9d96e8fd93cff7dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:32:07.849373 8505 tablet_replica.cc:439] T d07df80224814d59a0d5f104f13f68e0 P 7b178dcd80ae46949a1793f90f4a5cf9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } } }
I20260926 03:32:07.849704 8505 tablet_replica.cc:442] T d07df80224814d59a0d5f104f13f68e0 P 7b178dcd80ae46949a1793f90f4a5cf9: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:32:07.857612 8492 tablet_replica.cc:439] T d07df80224814d59a0d5f104f13f68e0 P 261504269b4c4c7e9d96e8fd93cff7dc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } } }
I20260926 03:32:07.857859 8492 tablet_replica.cc:442] T d07df80224814d59a0d5f104f13f68e0 P 261504269b4c4c7e9d96e8fd93cff7dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:32:07.867005 8521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:07.860275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54686 (local address 127.0.97.2:35309)
0926 03:32:07.860445 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:07.860460 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:07.861015 (+ 555us) server_negotiation.cc:407] Connection header received
0926 03:32:07.861176 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:07.861181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:07.861264 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:07.861355 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:32:07.861827 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:07.862985 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:07.864988 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:07.865636 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:07.865754 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:07.866226 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:32:07.866353 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:32:07.866444 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:07.866616 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 03:32:07.866697 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 03:32:07.884021 8521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:07.877615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50307 (local address 127.0.97.2:35309)
0926 03:32:07.877781 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:07.877788 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:07.878360 (+ 572us) server_negotiation.cc:407] Connection header received
0926 03:32:07.878619 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:07.878623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:07.878678 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:07.878759 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:32:07.879395 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:07.880244 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:07.882379 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:07.882801 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:07.882910 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:07.883391 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:32:07.883480 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:32:07.883541 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:07.883693 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 03:32:07.883754 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 03:32:07.884339 8529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:07.877503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35309 (local address 127.0.97.2:50307)
0926 03:32:07.878266 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:07.878290 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:07.878473 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:07.878793 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:07.878798 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:07.878811 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:32:07.879222 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:07.879234 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:07.880364 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:07.880369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:07.882236 (+ 1867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:07.882250 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:07.882987 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:07.882994 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:07.883166 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:07.883235 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:32:07.883568 (+ 333us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:32:07.883579 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 03:32:07.883669 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 03:32:07.883810 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":409,"threads_started":1}
I20260926 03:32:07.894315 8404 raft_consensus.cc:1261] T 6bdd021f80f448bda941975be737c605 P 261504269b4c4c7e9d96e8fd93cff7dc [term 1 FOLLOWER]: Refusing update from remote peer 7b178dcd80ae46949a1793f90f4a5cf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:32:07.894488 8127 raft_consensus.cc:1261] T 6bdd021f80f448bda941975be737c605 P b2b9ca83586547e18d54d7eb8eed7b76 [term 1 FOLLOWER]: Refusing update from remote peer 7b178dcd80ae46949a1793f90f4a5cf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:32:07.894958 8530 consensus_queue.cc:1035] T 6bdd021f80f448bda941975be737c605 P 7b178dcd80ae46949a1793f90f4a5cf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:32:07.895328 8505 consensus_queue.cc:1035] T 6bdd021f80f448bda941975be737c605 P 7b178dcd80ae46949a1793f90f4a5cf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:32:07.960299 7963 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:32:07.960536 7963 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:32:07.963461 7963 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dbe96762262e460388583b79167d2c1d: Sending DeleteTablet for 3 replicas of tablet 6bdd021f80f448bda941975be737c605
I20260926 03:32:07.964210 8102 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bdd021f80f448bda941975be737c605 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:32:07 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60342
I20260926 03:32:07.964350 8383 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bdd021f80f448bda941975be737c605 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:32:07 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54358
I20260926 03:32:07.964962 8241 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bdd021f80f448bda941975be737c605 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:32:07 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54662
I20260926 03:32:07.975327 8536 tablet_replica.cc:330] T 6bdd021f80f448bda941975be737c605 P 261504269b4c4c7e9d96e8fd93cff7dc: stopping tablet replica
I20260926 03:32:07.976104 8536 raft_consensus.cc:2229] T 6bdd021f80f448bda941975be737c605 P 261504269b4c4c7e9d96e8fd93cff7dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:32:07.976679 8536 raft_consensus.cc:2258] T 6bdd021f80f448bda941975be737c605 P 261504269b4c4c7e9d96e8fd93cff7dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:32:07.979692 8537 tablet_replica.cc:330] T 6bdd021f80f448bda941975be737c605 P 7b178dcd80ae46949a1793f90f4a5cf9: stopping tablet replica
I20260926 03:32:07.979934 8537 raft_consensus.cc:2229] T 6bdd021f80f448bda941975be737c605 P 7b178dcd80ae46949a1793f90f4a5cf9 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:32:07.980741 8537 raft_consensus.cc:2258] T 6bdd021f80f448bda941975be737c605 P 7b178dcd80ae46949a1793f90f4a5cf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:32:07.981038 8535 tablet_replica.cc:330] T 6bdd021f80f448bda941975be737c605 P b2b9ca83586547e18d54d7eb8eed7b76: stopping tablet replica
I20260926 03:32:07.981285 8535 raft_consensus.cc:2229] T 6bdd021f80f448bda941975be737c605 P b2b9ca83586547e18d54d7eb8eed7b76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:32:07.981554 8535 raft_consensus.cc:2258] T 6bdd021f80f448bda941975be737c605 P b2b9ca83586547e18d54d7eb8eed7b76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:32:07.981941 8537 ts_tablet_manager.cc:1905] T 6bdd021f80f448bda941975be737c605 P 7b178dcd80ae46949a1793f90f4a5cf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:32:07.982656 8535 ts_tablet_manager.cc:1905] T 6bdd021f80f448bda941975be737c605 P b2b9ca83586547e18d54d7eb8eed7b76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:32:07.984993 8536 ts_tablet_manager.cc:1905] T 6bdd021f80f448bda941975be737c605 P 261504269b4c4c7e9d96e8fd93cff7dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:32:07.989610 8535 ts_tablet_manager.cc:1918] T 6bdd021f80f448bda941975be737c605 P b2b9ca83586547e18d54d7eb8eed7b76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:32:07.989807 8535 log.cc:1196] T 6bdd021f80f448bda941975be737c605 P b2b9ca83586547e18d54d7eb8eed7b76: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-0/wal/wals/6bdd021f80f448bda941975be737c605
I20260926 03:32:07.990191 8537 ts_tablet_manager.cc:1918] T 6bdd021f80f448bda941975be737c605 P 7b178dcd80ae46949a1793f90f4a5cf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:32:07.990460 8537 log.cc:1196] T 6bdd021f80f448bda941975be737c605 P 7b178dcd80ae46949a1793f90f4a5cf9: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-1/wal/wals/6bdd021f80f448bda941975be737c605
I20260926 03:32:07.990504 8535 ts_tablet_manager.cc:1939] T 6bdd021f80f448bda941975be737c605 P b2b9ca83586547e18d54d7eb8eed7b76: Deleting consensus metadata
I20260926 03:32:07.990947 8537 ts_tablet_manager.cc:1939] T 6bdd021f80f448bda941975be737c605 P 7b178dcd80ae46949a1793f90f4a5cf9: Deleting consensus metadata
I20260926 03:32:07.991812 7952 catalog_manager.cc:4841] TS b2b9ca83586547e18d54d7eb8eed7b76 (127.0.97.1:46445): tablet 6bdd021f80f448bda941975be737c605 (table test-table [id=aec1060acd2941ef81353000f7ba5a0e]) successfully deleted
I20260926 03:32:07.992038 7951 catalog_manager.cc:4841] TS 7b178dcd80ae46949a1793f90f4a5cf9 (127.0.97.2:35309): tablet 6bdd021f80f448bda941975be737c605 (table test-table [id=aec1060acd2941ef81353000f7ba5a0e]) successfully deleted
I20260926 03:32:07.993964 8536 ts_tablet_manager.cc:1918] T 6bdd021f80f448bda941975be737c605 P 261504269b4c4c7e9d96e8fd93cff7dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:32:07.994089 8536 log.cc:1196] T 6bdd021f80f448bda941975be737c605 P 261504269b4c4c7e9d96e8fd93cff7dc: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-2/wal/wals/6bdd021f80f448bda941975be737c605
I20260926 03:32:07.994491 8536 ts_tablet_manager.cc:1939] T 6bdd021f80f448bda941975be737c605 P 261504269b4c4c7e9d96e8fd93cff7dc: Deleting consensus metadata
I20260926 03:32:07.995440 7951 catalog_manager.cc:4841] TS 261504269b4c4c7e9d96e8fd93cff7dc (127.0.97.3:43645): tablet 6bdd021f80f448bda941975be737c605 (table test-table [id=aec1060acd2941ef81353000f7ba5a0e]) successfully deleted
Sep 26 03:32:08 dist-test-slave-dt7p krb5kdc[7920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 03:32:08 dist-test-slave-dt7p krb5kdc[7920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:08 dist-test-slave-dt7p krb5kdc[7920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 03:32:08.034253 8512 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:08.022648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37088 (local address 127.0.97.62:33533)
0926 03:32:08.022905 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:08.022912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:08.023235 (+ 323us) server_negotiation.cc:407] Connection header received
0926 03:32:08.023414 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:08.023418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:08.023488 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:08.023572 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:08.025327 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:08.026078 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:08.027915 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:08.028327 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:08.028427 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:08.034007 (+ 5580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37088: BlockingRecv error: recv got EOF from 127.0.0.1:37088 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 26 03:32:08 dist-test-slave-dt7p krb5kdc[7920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:32:08.052538 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:08.037229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37092 (local address 127.0.97.62:33533)
0926 03:32:08.037457 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:08.037466 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:08.037964 (+ 498us) server_negotiation.cc:407] Connection header received
0926 03:32:08.038127 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:08.038133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:08.038210 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:08.038316 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:08.039383 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:08.040465 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:08.041795 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:08.042420 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:08.042560 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:08.046393 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:08.046400 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:08.046405 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:08.046447 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:08.049818 (+ 3371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:08.050414 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:08.050420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:08.050424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:08.050604 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:08.050942 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:08.050948 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:08.050953 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:08.051289 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:08.051559 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:08.051930 (+ 371us) server_negotiation.cc:299] Negotiation successful
0926 03:32:08.052172 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 03:32:08.056238 7963 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:32:08.056945 7963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:32:08.063480 8102 tablet_service.cc:1467] Processing CreateTablet for tablet d05a1b7574d74149b48f7cd779401e2b (DEFAULT_TABLE table=test-table [id=d2e11ffcf1d74b45be9f39e439b453d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:32:08.063932 8102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d05a1b7574d74149b48f7cd779401e2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:32:08.064097 8241 tablet_service.cc:1467] Processing CreateTablet for tablet d05a1b7574d74149b48f7cd779401e2b (DEFAULT_TABLE table=test-table [id=d2e11ffcf1d74b45be9f39e439b453d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:32:08.064424 8241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d05a1b7574d74149b48f7cd779401e2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:32:08.070012 8474 tablet_bootstrap.cc:492] T d05a1b7574d74149b48f7cd779401e2b P 7b178dcd80ae46949a1793f90f4a5cf9: Bootstrap starting.
I20260926 03:32:08.070267 8475 tablet_bootstrap.cc:492] T d05a1b7574d74149b48f7cd779401e2b P b2b9ca83586547e18d54d7eb8eed7b76: Bootstrap starting.
I20260926 03:32:08.070436 8383 tablet_service.cc:1467] Processing CreateTablet for tablet d05a1b7574d74149b48f7cd779401e2b (DEFAULT_TABLE table=test-table [id=d2e11ffcf1d74b45be9f39e439b453d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:32:08.070851 8383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d05a1b7574d74149b48f7cd779401e2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:32:08.071723 8474 tablet_bootstrap.cc:654] T d05a1b7574d74149b48f7cd779401e2b P 7b178dcd80ae46949a1793f90f4a5cf9: Neither blocks nor log segments found. Creating new log.
I20260926 03:32:08.071820 8475 tablet_bootstrap.cc:654] T d05a1b7574d74149b48f7cd779401e2b P b2b9ca83586547e18d54d7eb8eed7b76: Neither blocks nor log segments found. Creating new log.
I20260926 03:32:08.078871 8476 tablet_bootstrap.cc:492] T d05a1b7574d74149b48f7cd779401e2b P 261504269b4c4c7e9d96e8fd93cff7dc: Bootstrap starting.
I20260926 03:32:08.081070 8476 tablet_bootstrap.cc:654] T d05a1b7574d74149b48f7cd779401e2b P 261504269b4c4c7e9d96e8fd93cff7dc: Neither blocks nor log segments found. Creating new log.
I20260926 03:32:08.090771 8476 tablet_bootstrap.cc:492] T d05a1b7574d74149b48f7cd779401e2b P 261504269b4c4c7e9d96e8fd93cff7dc: No bootstrap required, opened a new log
I20260926 03:32:08.091017 8476 ts_tablet_manager.cc:1397] T d05a1b7574d74149b48f7cd779401e2b P 261504269b4c4c7e9d96e8fd93cff7dc: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260926 03:32:08.092182 8476 raft_consensus.cc:348] T d05a1b7574d74149b48f7cd779401e2b P 261504269b4c4c7e9d96e8fd93cff7dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } }
I20260926 03:32:08.092674 8476 raft_consensus.cc:374] T d05a1b7574d74149b48f7cd779401e2b P 261504269b4c4c7e9d96e8fd93cff7dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:32:08.092841 8476 raft_consensus.cc:729] T d05a1b7574d74149b48f7cd779401e2b P 261504269b4c4c7e9d96e8fd93cff7dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 261504269b4c4c7e9d96e8fd93cff7dc, State: Initialized, Role: FOLLOWER
I20260926 03:32:08.093235 8476 consensus_queue.cc:260] T d05a1b7574d74149b48f7cd779401e2b P 261504269b4c4c7e9d96e8fd93cff7dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } }
I20260926 03:32:08.097121 8476 ts_tablet_manager.cc:1428] T d05a1b7574d74149b48f7cd779401e2b P 261504269b4c4c7e9d96e8fd93cff7dc: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20260926 03:32:08.101243 8475 tablet_bootstrap.cc:492] T d05a1b7574d74149b48f7cd779401e2b P b2b9ca83586547e18d54d7eb8eed7b76: No bootstrap required, opened a new log
I20260926 03:32:08.101411 8475 ts_tablet_manager.cc:1397] T d05a1b7574d74149b48f7cd779401e2b P b2b9ca83586547e18d54d7eb8eed7b76: Time spent bootstrapping tablet: real 0.031s user 0.007s sys 0.022s
I20260926 03:32:08.101800 8474 tablet_bootstrap.cc:492] T d05a1b7574d74149b48f7cd779401e2b P 7b178dcd80ae46949a1793f90f4a5cf9: No bootstrap required, opened a new log
I20260926 03:32:08.101982 8474 ts_tablet_manager.cc:1397] T d05a1b7574d74149b48f7cd779401e2b P 7b178dcd80ae46949a1793f90f4a5cf9: Time spent bootstrapping tablet: real 0.032s user 0.008s sys 0.017s
I20260926 03:32:08.102213 8475 raft_consensus.cc:348] T d05a1b7574d74149b48f7cd779401e2b P b2b9ca83586547e18d54d7eb8eed7b76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } }
I20260926 03:32:08.102481 8475 raft_consensus.cc:374] T d05a1b7574d74149b48f7cd779401e2b P b2b9ca83586547e18d54d7eb8eed7b76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:32:08.102592 8475 raft_consensus.cc:729] T d05a1b7574d74149b48f7cd779401e2b P b2b9ca83586547e18d54d7eb8eed7b76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2b9ca83586547e18d54d7eb8eed7b76, State: Initialized, Role: FOLLOWER
I20260926 03:32:08.102748 8474 raft_consensus.cc:348] T d05a1b7574d74149b48f7cd779401e2b P 7b178dcd80ae46949a1793f90f4a5cf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } }
I20260926 03:32:08.102862 8475 consensus_queue.cc:260] T d05a1b7574d74149b48f7cd779401e2b P b2b9ca83586547e18d54d7eb8eed7b76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } }
I20260926 03:32:08.102986 8474 raft_consensus.cc:374] T d05a1b7574d74149b48f7cd779401e2b P 7b178dcd80ae46949a1793f90f4a5cf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:32:08.103087 8474 raft_consensus.cc:729] T d05a1b7574d74149b48f7cd779401e2b P 7b178dcd80ae46949a1793f90f4a5cf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b178dcd80ae46949a1793f90f4a5cf9, State: Initialized, Role: FOLLOWER
I20260926 03:32:08.103325 8474 consensus_queue.cc:260] T d05a1b7574d74149b48f7cd779401e2b P 7b178dcd80ae46949a1793f90f4a5cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } }
I20260926 03:32:08.103809 8474 ts_tablet_manager.cc:1428] T d05a1b7574d74149b48f7cd779401e2b P 7b178dcd80ae46949a1793f90f4a5cf9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 03:32:08.104598 8475 ts_tablet_manager.cc:1428] T d05a1b7574d74149b48f7cd779401e2b P b2b9ca83586547e18d54d7eb8eed7b76: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 03:32:08.333133 8492 raft_consensus.cc:482] T d05a1b7574d74149b48f7cd779401e2b P 261504269b4c4c7e9d96e8fd93cff7dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:32:08.333364 8492 raft_consensus.cc:504] T d05a1b7574d74149b48f7cd779401e2b P 261504269b4c4c7e9d96e8fd93cff7dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } }
I20260926 03:32:08.334102 8492 leader_election.cc:288] T d05a1b7574d74149b48f7cd779401e2b P 261504269b4c4c7e9d96e8fd93cff7dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2b9ca83586547e18d54d7eb8eed7b76 (127.0.97.1:46445), 7b178dcd80ae46949a1793f90f4a5cf9 (127.0.97.2:35309)
I20260926 03:32:08.334707 8127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d05a1b7574d74149b48f7cd779401e2b" candidate_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" is_pre_election: true
I20260926 03:32:08.334743 8261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d05a1b7574d74149b48f7cd779401e2b" candidate_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" is_pre_election: true
I20260926 03:32:08.334968 8261 raft_consensus.cc:2454] T d05a1b7574d74149b48f7cd779401e2b P 7b178dcd80ae46949a1793f90f4a5cf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 261504269b4c4c7e9d96e8fd93cff7dc in term 0.
I20260926 03:32:08.334964 8127 raft_consensus.cc:2454] T d05a1b7574d74149b48f7cd779401e2b P b2b9ca83586547e18d54d7eb8eed7b76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 261504269b4c4c7e9d96e8fd93cff7dc in term 0.
I20260926 03:32:08.335479 8333 leader_election.cc:302] T d05a1b7574d74149b48f7cd779401e2b P 261504269b4c4c7e9d96e8fd93cff7dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 261504269b4c4c7e9d96e8fd93cff7dc, 7b178dcd80ae46949a1793f90f4a5cf9; no voters:
I20260926 03:32:08.335778 8492 raft_consensus.cc:2781] T d05a1b7574d74149b48f7cd779401e2b P 261504269b4c4c7e9d96e8fd93cff7dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:32:08.335897 8492 raft_consensus.cc:482] T d05a1b7574d74149b48f7cd779401e2b P 261504269b4c4c7e9d96e8fd93cff7dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:32:08.335987 8492 raft_consensus.cc:3037] T d05a1b7574d74149b48f7cd779401e2b P 261504269b4c4c7e9d96e8fd93cff7dc [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:32:08.338975 8492 raft_consensus.cc:504] T d05a1b7574d74149b48f7cd779401e2b P 261504269b4c4c7e9d96e8fd93cff7dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } }
I20260926 03:32:08.339686 8492 leader_election.cc:288] T d05a1b7574d74149b48f7cd779401e2b P 261504269b4c4c7e9d96e8fd93cff7dc [CANDIDATE]: Term 1 election: Requested vote from peers b2b9ca83586547e18d54d7eb8eed7b76 (127.0.97.1:46445), 7b178dcd80ae46949a1793f90f4a5cf9 (127.0.97.2:35309)
I20260926 03:32:08.340216 8127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d05a1b7574d74149b48f7cd779401e2b" candidate_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2b9ca83586547e18d54d7eb8eed7b76"
I20260926 03:32:08.340257 8261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d05a1b7574d74149b48f7cd779401e2b" candidate_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b178dcd80ae46949a1793f90f4a5cf9"
I20260926 03:32:08.340437 8127 raft_consensus.cc:3037] T d05a1b7574d74149b48f7cd779401e2b P b2b9ca83586547e18d54d7eb8eed7b76 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:32:08.340502 8261 raft_consensus.cc:3037] T d05a1b7574d74149b48f7cd779401e2b P 7b178dcd80ae46949a1793f90f4a5cf9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:32:08.342751 8127 raft_consensus.cc:2454] T d05a1b7574d74149b48f7cd779401e2b P b2b9ca83586547e18d54d7eb8eed7b76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 261504269b4c4c7e9d96e8fd93cff7dc in term 1.
I20260926 03:32:08.343228 8334 leader_election.cc:302] T d05a1b7574d74149b48f7cd779401e2b P 261504269b4c4c7e9d96e8fd93cff7dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 261504269b4c4c7e9d96e8fd93cff7dc, b2b9ca83586547e18d54d7eb8eed7b76; no voters:
I20260926 03:32:08.343516 8492 raft_consensus.cc:2781] T d05a1b7574d74149b48f7cd779401e2b P 261504269b4c4c7e9d96e8fd93cff7dc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:32:08.343637 8261 raft_consensus.cc:2454] T d05a1b7574d74149b48f7cd779401e2b P 7b178dcd80ae46949a1793f90f4a5cf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 261504269b4c4c7e9d96e8fd93cff7dc in term 1.
I20260926 03:32:08.344063 8492 raft_consensus.cc:686] T d05a1b7574d74149b48f7cd779401e2b P 261504269b4c4c7e9d96e8fd93cff7dc [term 1 LEADER]: Becoming Leader. State: Replica: 261504269b4c4c7e9d96e8fd93cff7dc, State: Running, Role: LEADER
I20260926 03:32:08.344359 8492 consensus_queue.cc:237] T d05a1b7574d74149b48f7cd779401e2b P 261504269b4c4c7e9d96e8fd93cff7dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } }
I20260926 03:32:08.347100 7964 catalog_manager.cc:5495] T d05a1b7574d74149b48f7cd779401e2b P 261504269b4c4c7e9d96e8fd93cff7dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 261504269b4c4c7e9d96e8fd93cff7dc (127.0.97.3). New cstate: current_term: 1 leader_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } health_report { overall_health: HEALTHY } } }
I20260926 03:32:08.373452 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:08.367229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54372 (local address 127.0.97.3:43645)
0926 03:32:08.367776 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:08.367781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:08.367797 (+ 16us) server_negotiation.cc:407] Connection header received
0926 03:32:08.367852 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:08.367856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:08.367916 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:08.368000 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:32:08.368420 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:08.369401 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:08.371385 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:08.371829 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:08.371963 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:08.372478 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:32:08.372688 (+ 210us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:32:08.372767 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:08.372988 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 03:32:08.373063 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":297,"threads_started":1}
I20260926 03:32:08.376298 8127 raft_consensus.cc:1261] T d05a1b7574d74149b48f7cd779401e2b P b2b9ca83586547e18d54d7eb8eed7b76 [term 1 FOLLOWER]: Refusing update from remote peer 261504269b4c4c7e9d96e8fd93cff7dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:32:08.376302 8261 raft_consensus.cc:1261] T d05a1b7574d74149b48f7cd779401e2b P 7b178dcd80ae46949a1793f90f4a5cf9 [term 1 FOLLOWER]: Refusing update from remote peer 261504269b4c4c7e9d96e8fd93cff7dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:32:08.376940 8553 consensus_queue.cc:1035] T d05a1b7574d74149b48f7cd779401e2b P 261504269b4c4c7e9d96e8fd93cff7dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:32:08.377262 8492 consensus_queue.cc:1035] T d05a1b7574d74149b48f7cd779401e2b P 261504269b4c4c7e9d96e8fd93cff7dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:32:08.389638 7964 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:32:08.389909 7964 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:32:08.393991 7964 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dbe96762262e460388583b79167d2c1d: Sending DeleteTablet for 3 replicas of tablet d05a1b7574d74149b48f7cd779401e2b
I20260926 03:32:08.394956 8241 tablet_service.cc:1514] Processing DeleteTablet for tablet d05a1b7574d74149b48f7cd779401e2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:32:08 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54662
I20260926 03:32:08.395268 8383 tablet_service.cc:1514] Processing DeleteTablet for tablet d05a1b7574d74149b48f7cd779401e2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:32:08 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54358
I20260926 03:32:08.395294 8102 tablet_service.cc:1514] Processing DeleteTablet for tablet d05a1b7574d74149b48f7cd779401e2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:32:08 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60342
I20260926 03:32:08.395555 8537 tablet_replica.cc:330] T d05a1b7574d74149b48f7cd779401e2b P 7b178dcd80ae46949a1793f90f4a5cf9: stopping tablet replica
I20260926 03:32:08.395579 8379 tablet_service.cc:1514] Processing DeleteTablet for tablet d05a1b7574d74149b48f7cd779401e2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:32:08 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54358
I20260926 03:32:08.395779 8537 raft_consensus.cc:2229] T d05a1b7574d74149b48f7cd779401e2b P 7b178dcd80ae46949a1793f90f4a5cf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:32:08.395833 8536 tablet_replica.cc:330] T d05a1b7574d74149b48f7cd779401e2b P 261504269b4c4c7e9d96e8fd93cff7dc: stopping tablet replica
I20260926 03:32:08.395962 8537 raft_consensus.cc:2258] T d05a1b7574d74149b48f7cd779401e2b P 7b178dcd80ae46949a1793f90f4a5cf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:32:08.396014 8536 raft_consensus.cc:2229] T d05a1b7574d74149b48f7cd779401e2b P 261504269b4c4c7e9d96e8fd93cff7dc [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:32:08.396304 8536 raft_consensus.cc:2258] T d05a1b7574d74149b48f7cd779401e2b P 261504269b4c4c7e9d96e8fd93cff7dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:32:08.396588 8535 tablet_replica.cc:330] T d05a1b7574d74149b48f7cd779401e2b P b2b9ca83586547e18d54d7eb8eed7b76: stopping tablet replica
I20260926 03:32:08.396773 8535 raft_consensus.cc:2229] T d05a1b7574d74149b48f7cd779401e2b P b2b9ca83586547e18d54d7eb8eed7b76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:32:08.396971 8535 raft_consensus.cc:2258] T d05a1b7574d74149b48f7cd779401e2b P b2b9ca83586547e18d54d7eb8eed7b76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:32:08.397390 8537 ts_tablet_manager.cc:1905] T d05a1b7574d74149b48f7cd779401e2b P 7b178dcd80ae46949a1793f90f4a5cf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:32:08.397219 7963 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:32:08.397830 7963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:32:08.398183 8536 ts_tablet_manager.cc:1905] T d05a1b7574d74149b48f7cd779401e2b P 261504269b4c4c7e9d96e8fd93cff7dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:32:08.402351 8535 ts_tablet_manager.cc:1905] T d05a1b7574d74149b48f7cd779401e2b P b2b9ca83586547e18d54d7eb8eed7b76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:32:08.404383 8537 ts_tablet_manager.cc:1918] T d05a1b7574d74149b48f7cd779401e2b P 7b178dcd80ae46949a1793f90f4a5cf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:32:08.404547 8537 log.cc:1196] T d05a1b7574d74149b48f7cd779401e2b P 7b178dcd80ae46949a1793f90f4a5cf9: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-1/wal/wals/d05a1b7574d74149b48f7cd779401e2b
I20260926 03:32:08.405054 8537 ts_tablet_manager.cc:1939] T d05a1b7574d74149b48f7cd779401e2b P 7b178dcd80ae46949a1793f90f4a5cf9: Deleting consensus metadata
I20260926 03:32:08.405925 8536 ts_tablet_manager.cc:1918] T d05a1b7574d74149b48f7cd779401e2b P 261504269b4c4c7e9d96e8fd93cff7dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:32:08.406241 7951 catalog_manager.cc:4841] TS 7b178dcd80ae46949a1793f90f4a5cf9 (127.0.97.2:35309): tablet d05a1b7574d74149b48f7cd779401e2b (table test-table [id=d2e11ffcf1d74b45be9f39e439b453d3]) successfully deleted
I20260926 03:32:08.406128 8536 log.cc:1196] T d05a1b7574d74149b48f7cd779401e2b P 261504269b4c4c7e9d96e8fd93cff7dc: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-2/wal/wals/d05a1b7574d74149b48f7cd779401e2b
I20260926 03:32:08.407755 8536 ts_tablet_manager.cc:1939] T d05a1b7574d74149b48f7cd779401e2b P 261504269b4c4c7e9d96e8fd93cff7dc: Deleting consensus metadata
I20260926 03:32:08.408761 7951 catalog_manager.cc:4841] TS 261504269b4c4c7e9d96e8fd93cff7dc (127.0.97.3:43645): tablet d05a1b7574d74149b48f7cd779401e2b (table test-table [id=d2e11ffcf1d74b45be9f39e439b453d3]) successfully deleted
W20260926 03:32:08.409263 7951 catalog_manager.cc:4808] TS 261504269b4c4c7e9d96e8fd93cff7dc (127.0.97.3:43645): delete failed for tablet d05a1b7574d74149b48f7cd779401e2b because the tablet was not found. No further retry: Not found: Tablet not found: d05a1b7574d74149b48f7cd779401e2b
I20260926 03:32:08.411248 8535 ts_tablet_manager.cc:1918] T d05a1b7574d74149b48f7cd779401e2b P b2b9ca83586547e18d54d7eb8eed7b76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:32:08.411397 8535 log.cc:1196] T d05a1b7574d74149b48f7cd779401e2b P b2b9ca83586547e18d54d7eb8eed7b76: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-0/wal/wals/d05a1b7574d74149b48f7cd779401e2b
I20260926 03:32:08.411772 8535 ts_tablet_manager.cc:1939] T d05a1b7574d74149b48f7cd779401e2b P b2b9ca83586547e18d54d7eb8eed7b76: Deleting consensus metadata
I20260926 03:32:08.412583 7952 catalog_manager.cc:4841] TS b2b9ca83586547e18d54d7eb8eed7b76 (127.0.97.1:46445): tablet d05a1b7574d74149b48f7cd779401e2b (table test-table [id=d2e11ffcf1d74b45be9f39e439b453d3]) successfully deleted
I20260926 03:32:08.414039 8102 tablet_service.cc:1467] Processing CreateTablet for tablet 3346839176924292b8ff50cbe7bdc15f (DEFAULT_TABLE table=test-table [id=4e9007da72fe4ea1afe671b3144e1ebf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:32:08.414371 8241 tablet_service.cc:1467] Processing CreateTablet for tablet 3346839176924292b8ff50cbe7bdc15f (DEFAULT_TABLE table=test-table [id=4e9007da72fe4ea1afe671b3144e1ebf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:32:08.414440 8102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3346839176924292b8ff50cbe7bdc15f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:32:08.414775 8241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3346839176924292b8ff50cbe7bdc15f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:32:08.416916 8379 tablet_service.cc:1467] Processing CreateTablet for tablet 3346839176924292b8ff50cbe7bdc15f (DEFAULT_TABLE table=test-table [id=4e9007da72fe4ea1afe671b3144e1ebf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:32:08.417318 8379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3346839176924292b8ff50cbe7bdc15f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:32:08.420127 8474 tablet_bootstrap.cc:492] T 3346839176924292b8ff50cbe7bdc15f P 7b178dcd80ae46949a1793f90f4a5cf9: Bootstrap starting.
I20260926 03:32:08.420526 8475 tablet_bootstrap.cc:492] T 3346839176924292b8ff50cbe7bdc15f P b2b9ca83586547e18d54d7eb8eed7b76: Bootstrap starting.
I20260926 03:32:08.421916 8474 tablet_bootstrap.cc:654] T 3346839176924292b8ff50cbe7bdc15f P 7b178dcd80ae46949a1793f90f4a5cf9: Neither blocks nor log segments found. Creating new log.
I20260926 03:32:08.422058 8475 tablet_bootstrap.cc:654] T 3346839176924292b8ff50cbe7bdc15f P b2b9ca83586547e18d54d7eb8eed7b76: Neither blocks nor log segments found. Creating new log.
I20260926 03:32:08.424347 8476 tablet_bootstrap.cc:492] T 3346839176924292b8ff50cbe7bdc15f P 261504269b4c4c7e9d96e8fd93cff7dc: Bootstrap starting.
I20260926 03:32:08.425875 8476 tablet_bootstrap.cc:654] T 3346839176924292b8ff50cbe7bdc15f P 261504269b4c4c7e9d96e8fd93cff7dc: Neither blocks nor log segments found. Creating new log.
I20260926 03:32:08.431535 8476 tablet_bootstrap.cc:492] T 3346839176924292b8ff50cbe7bdc15f P 261504269b4c4c7e9d96e8fd93cff7dc: No bootstrap required, opened a new log
I20260926 03:32:08.431726 8476 ts_tablet_manager.cc:1397] T 3346839176924292b8ff50cbe7bdc15f P 261504269b4c4c7e9d96e8fd93cff7dc: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260926 03:32:08.432497 8475 tablet_bootstrap.cc:492] T 3346839176924292b8ff50cbe7bdc15f P b2b9ca83586547e18d54d7eb8eed7b76: No bootstrap required, opened a new log
I20260926 03:32:08.432777 8475 ts_tablet_manager.cc:1397] T 3346839176924292b8ff50cbe7bdc15f P b2b9ca83586547e18d54d7eb8eed7b76: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.010s
I20260926 03:32:08.432636 8476 raft_consensus.cc:348] T 3346839176924292b8ff50cbe7bdc15f P 261504269b4c4c7e9d96e8fd93cff7dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } }
I20260926 03:32:08.432889 8476 raft_consensus.cc:374] T 3346839176924292b8ff50cbe7bdc15f P 261504269b4c4c7e9d96e8fd93cff7dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:32:08.432978 8476 raft_consensus.cc:729] T 3346839176924292b8ff50cbe7bdc15f P 261504269b4c4c7e9d96e8fd93cff7dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 261504269b4c4c7e9d96e8fd93cff7dc, State: Initialized, Role: FOLLOWER
I20260926 03:32:08.433259 8476 consensus_queue.cc:260] T 3346839176924292b8ff50cbe7bdc15f P 261504269b4c4c7e9d96e8fd93cff7dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } }
I20260926 03:32:08.433908 8476 ts_tablet_manager.cc:1428] T 3346839176924292b8ff50cbe7bdc15f P 261504269b4c4c7e9d96e8fd93cff7dc: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 03:32:08.434072 8474 tablet_bootstrap.cc:492] T 3346839176924292b8ff50cbe7bdc15f P 7b178dcd80ae46949a1793f90f4a5cf9: No bootstrap required, opened a new log
I20260926 03:32:08.434245 8474 ts_tablet_manager.cc:1397] T 3346839176924292b8ff50cbe7bdc15f P 7b178dcd80ae46949a1793f90f4a5cf9: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.001s
I20260926 03:32:08.435010 8474 raft_consensus.cc:348] T 3346839176924292b8ff50cbe7bdc15f P 7b178dcd80ae46949a1793f90f4a5cf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } }
I20260926 03:32:08.435298 8474 raft_consensus.cc:374] T 3346839176924292b8ff50cbe7bdc15f P 7b178dcd80ae46949a1793f90f4a5cf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:32:08.435410 8474 raft_consensus.cc:729] T 3346839176924292b8ff50cbe7bdc15f P 7b178dcd80ae46949a1793f90f4a5cf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b178dcd80ae46949a1793f90f4a5cf9, State: Initialized, Role: FOLLOWER
I20260926 03:32:08.435684 8474 consensus_queue.cc:260] T 3346839176924292b8ff50cbe7bdc15f P 7b178dcd80ae46949a1793f90f4a5cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } }
I20260926 03:32:08.436157 8474 ts_tablet_manager.cc:1428] T 3346839176924292b8ff50cbe7bdc15f P 7b178dcd80ae46949a1793f90f4a5cf9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 03:32:08.436820 8475 raft_consensus.cc:348] T 3346839176924292b8ff50cbe7bdc15f P b2b9ca83586547e18d54d7eb8eed7b76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } }
I20260926 03:32:08.437071 8475 raft_consensus.cc:374] T 3346839176924292b8ff50cbe7bdc15f P b2b9ca83586547e18d54d7eb8eed7b76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:32:08.437367 8475 raft_consensus.cc:729] T 3346839176924292b8ff50cbe7bdc15f P b2b9ca83586547e18d54d7eb8eed7b76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2b9ca83586547e18d54d7eb8eed7b76, State: Initialized, Role: FOLLOWER
I20260926 03:32:08.437662 8475 consensus_queue.cc:260] T 3346839176924292b8ff50cbe7bdc15f P b2b9ca83586547e18d54d7eb8eed7b76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } }
I20260926 03:32:08.438166 8475 ts_tablet_manager.cc:1428] T 3346839176924292b8ff50cbe7bdc15f P b2b9ca83586547e18d54d7eb8eed7b76: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260926 03:32:08.598335 8559 raft_consensus.cc:482] T 3346839176924292b8ff50cbe7bdc15f P 261504269b4c4c7e9d96e8fd93cff7dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:32:08.598606 8559 raft_consensus.cc:504] T 3346839176924292b8ff50cbe7bdc15f P 261504269b4c4c7e9d96e8fd93cff7dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } }
I20260926 03:32:08.599435 8559 leader_election.cc:288] T 3346839176924292b8ff50cbe7bdc15f P 261504269b4c4c7e9d96e8fd93cff7dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2b9ca83586547e18d54d7eb8eed7b76 (127.0.97.1:46445), 7b178dcd80ae46949a1793f90f4a5cf9 (127.0.97.2:35309)
I20260926 03:32:08.600062 8127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3346839176924292b8ff50cbe7bdc15f" candidate_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" is_pre_election: true
I20260926 03:32:08.600080 8261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3346839176924292b8ff50cbe7bdc15f" candidate_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" is_pre_election: true
I20260926 03:32:08.600319 8127 raft_consensus.cc:2454] T 3346839176924292b8ff50cbe7bdc15f P b2b9ca83586547e18d54d7eb8eed7b76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 261504269b4c4c7e9d96e8fd93cff7dc in term 0.
I20260926 03:32:08.600324 8261 raft_consensus.cc:2454] T 3346839176924292b8ff50cbe7bdc15f P 7b178dcd80ae46949a1793f90f4a5cf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 261504269b4c4c7e9d96e8fd93cff7dc in term 0.
I20260926 03:32:08.600785 8334 leader_election.cc:302] T 3346839176924292b8ff50cbe7bdc15f P 261504269b4c4c7e9d96e8fd93cff7dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 261504269b4c4c7e9d96e8fd93cff7dc, b2b9ca83586547e18d54d7eb8eed7b76; no voters:
I20260926 03:32:08.601054 8559 raft_consensus.cc:2781] T 3346839176924292b8ff50cbe7bdc15f P 261504269b4c4c7e9d96e8fd93cff7dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:32:08.601199 8559 raft_consensus.cc:482] T 3346839176924292b8ff50cbe7bdc15f P 261504269b4c4c7e9d96e8fd93cff7dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:32:08.601297 8559 raft_consensus.cc:3037] T 3346839176924292b8ff50cbe7bdc15f P 261504269b4c4c7e9d96e8fd93cff7dc [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:32:08.603276 8559 raft_consensus.cc:504] T 3346839176924292b8ff50cbe7bdc15f P 261504269b4c4c7e9d96e8fd93cff7dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } }
I20260926 03:32:08.603883 8559 leader_election.cc:288] T 3346839176924292b8ff50cbe7bdc15f P 261504269b4c4c7e9d96e8fd93cff7dc [CANDIDATE]: Term 1 election: Requested vote from peers b2b9ca83586547e18d54d7eb8eed7b76 (127.0.97.1:46445), 7b178dcd80ae46949a1793f90f4a5cf9 (127.0.97.2:35309)
I20260926 03:32:08.604292 8127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3346839176924292b8ff50cbe7bdc15f" candidate_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2b9ca83586547e18d54d7eb8eed7b76"
I20260926 03:32:08.604503 8127 raft_consensus.cc:3037] T 3346839176924292b8ff50cbe7bdc15f P b2b9ca83586547e18d54d7eb8eed7b76 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:32:08.604486 8261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3346839176924292b8ff50cbe7bdc15f" candidate_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b178dcd80ae46949a1793f90f4a5cf9"
I20260926 03:32:08.604727 8261 raft_consensus.cc:3037] T 3346839176924292b8ff50cbe7bdc15f P 7b178dcd80ae46949a1793f90f4a5cf9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:32:08.607564 8127 raft_consensus.cc:2454] T 3346839176924292b8ff50cbe7bdc15f P b2b9ca83586547e18d54d7eb8eed7b76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 261504269b4c4c7e9d96e8fd93cff7dc in term 1.
I20260926 03:32:08.607952 8261 raft_consensus.cc:2454] T 3346839176924292b8ff50cbe7bdc15f P 7b178dcd80ae46949a1793f90f4a5cf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 261504269b4c4c7e9d96e8fd93cff7dc in term 1.
I20260926 03:32:08.608057 8334 leader_election.cc:302] T 3346839176924292b8ff50cbe7bdc15f P 261504269b4c4c7e9d96e8fd93cff7dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 261504269b4c4c7e9d96e8fd93cff7dc, b2b9ca83586547e18d54d7eb8eed7b76; no voters:
I20260926 03:32:08.608356 8559 raft_consensus.cc:2781] T 3346839176924292b8ff50cbe7bdc15f P 261504269b4c4c7e9d96e8fd93cff7dc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:32:08.608541 8559 raft_consensus.cc:686] T 3346839176924292b8ff50cbe7bdc15f P 261504269b4c4c7e9d96e8fd93cff7dc [term 1 LEADER]: Becoming Leader. State: Replica: 261504269b4c4c7e9d96e8fd93cff7dc, State: Running, Role: LEADER
I20260926 03:32:08.608837 8559 consensus_queue.cc:237] T 3346839176924292b8ff50cbe7bdc15f P 261504269b4c4c7e9d96e8fd93cff7dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } }
I20260926 03:32:08.611788 7964 catalog_manager.cc:5495] T 3346839176924292b8ff50cbe7bdc15f P 261504269b4c4c7e9d96e8fd93cff7dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 261504269b4c4c7e9d96e8fd93cff7dc (127.0.97.3). New cstate: current_term: 1 leader_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "261504269b4c4c7e9d96e8fd93cff7dc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43645 } health_report { overall_health: HEALTHY } } }
I20260926 03:32:08.648931 8567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:08.639316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35309 (local address 127.0.97.3:58283)
0926 03:32:08.640046 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:08.640074 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:08.640244 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:08.643565 (+ 3321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:08.643568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:08.643579 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:32:08.643857 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:08.643865 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:08.645534 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:08.645538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:08.646665 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:08.646677 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:08.647699 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:08.647703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:08.647817 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:08.647877 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:32:08.648349 (+ 472us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:32:08.648357 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 03:32:08.648415 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 03:32:08.648638 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":264,"threads_started":1}
I20260926 03:32:08.648989 8568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:08.639411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58283 (local address 127.0.97.2:35309)
0926 03:32:08.643202 (+ 3791us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:08.643209 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:08.643230 (+ 21us) server_negotiation.cc:407] Connection header received
0926 03:32:08.643314 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:08.643320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:08.643417 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:08.643523 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:32:08.644015 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:08.645373 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:08.646816 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:08.647537 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:08.647655 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:08.648107 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:32:08.648242 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:32:08.648330 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:08.648489 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 03:32:08.648589 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3629,"thread_start_us":3458,"threads_started":1}
I20260926 03:32:08.657613 8127 raft_consensus.cc:1261] T 3346839176924292b8ff50cbe7bdc15f P b2b9ca83586547e18d54d7eb8eed7b76 [term 1 FOLLOWER]: Refusing update from remote peer 261504269b4c4c7e9d96e8fd93cff7dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:32:08.657850 8261 raft_consensus.cc:1261] T 3346839176924292b8ff50cbe7bdc15f P 7b178dcd80ae46949a1793f90f4a5cf9 [term 1 FOLLOWER]: Refusing update from remote peer 261504269b4c4c7e9d96e8fd93cff7dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:32:08.658339 8553 consensus_queue.cc:1035] T 3346839176924292b8ff50cbe7bdc15f P 261504269b4c4c7e9d96e8fd93cff7dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2b9ca83586547e18d54d7eb8eed7b76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:32:08.658600 8559 consensus_queue.cc:1035] T 3346839176924292b8ff50cbe7bdc15f P 261504269b4c4c7e9d96e8fd93cff7dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b178dcd80ae46949a1793f90f4a5cf9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:32:08.688668 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:08.682994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35007 (local address 127.0.97.3:43645)
0926 03:32:08.683280 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:08.683288 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:08.683508 (+ 220us) server_negotiation.cc:407] Connection header received
0926 03:32:08.683649 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:08.683652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:08.683710 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:08.683778 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:32:08.684264 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:08.685189 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:08.687124 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:08.687531 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:08.687629 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:08.688049 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:32:08.688145 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:32:08.688217 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:08.688342 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 03:32:08.688404 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260926 03:32:08.689118 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:08.682860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43645 (local address 127.0.97.2:35007)
0926 03:32:08.683394 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:08.683413 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:08.683554 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:08.683812 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:08.683816 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:08.683827 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:32:08.684115 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:08.684125 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:08.685312 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:08.685319 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:08.686991 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:08.687008 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:08.687715 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:08.687720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:08.687844 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:08.687881 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:32:08.688241 (+ 360us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:32:08.688249 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 03:32:08.688314 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 03:32:08.688411 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":431,"threads_started":1}
I20260926 03:32:08.716548 7963 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:32:08.716835 7963 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:32:08.719650 7963 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dbe96762262e460388583b79167d2c1d: Sending DeleteTablet for 3 replicas of tablet 3346839176924292b8ff50cbe7bdc15f
I20260926 03:32:08.720412 8102 tablet_service.cc:1514] Processing DeleteTablet for tablet 3346839176924292b8ff50cbe7bdc15f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:32:08 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60342
I20260926 03:32:08.720716 8379 tablet_service.cc:1514] Processing DeleteTablet for tablet 3346839176924292b8ff50cbe7bdc15f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:32:08 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54358
I20260926 03:32:08.720834 8535 tablet_replica.cc:330] T 3346839176924292b8ff50cbe7bdc15f P b2b9ca83586547e18d54d7eb8eed7b76: stopping tablet replica
I20260926 03:32:08.720783 8241 tablet_service.cc:1514] Processing DeleteTablet for tablet 3346839176924292b8ff50cbe7bdc15f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:32:08 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54662
I20260926 03:32:08.721010 8536 tablet_replica.cc:330] T 3346839176924292b8ff50cbe7bdc15f P 261504269b4c4c7e9d96e8fd93cff7dc: stopping tablet replica
I20260926 03:32:08.721019 8535 raft_consensus.cc:2229] T 3346839176924292b8ff50cbe7bdc15f P b2b9ca83586547e18d54d7eb8eed7b76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:32:08.721098 8537 tablet_replica.cc:330] T 3346839176924292b8ff50cbe7bdc15f P 7b178dcd80ae46949a1793f90f4a5cf9: stopping tablet replica
I20260926 03:32:08.721201 8536 raft_consensus.cc:2229] T 3346839176924292b8ff50cbe7bdc15f P 261504269b4c4c7e9d96e8fd93cff7dc [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:32:08.721232 8535 raft_consensus.cc:2258] T 3346839176924292b8ff50cbe7bdc15f P b2b9ca83586547e18d54d7eb8eed7b76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:32:08.721277 8537 raft_consensus.cc:2229] T 3346839176924292b8ff50cbe7bdc15f P 7b178dcd80ae46949a1793f90f4a5cf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:32:08.721465 8537 raft_consensus.cc:2258] T 3346839176924292b8ff50cbe7bdc15f P 7b178dcd80ae46949a1793f90f4a5cf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:32:08.721558 8536 raft_consensus.cc:2258] T 3346839176924292b8ff50cbe7bdc15f P 261504269b4c4c7e9d96e8fd93cff7dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:32:08.722528 8535 ts_tablet_manager.cc:1905] T 3346839176924292b8ff50cbe7bdc15f P b2b9ca83586547e18d54d7eb8eed7b76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:32:08.722841 8536 ts_tablet_manager.cc:1905] T 3346839176924292b8ff50cbe7bdc15f P 261504269b4c4c7e9d96e8fd93cff7dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:32:08.722874 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 8036
I20260926 03:32:08.724242 8537 ts_tablet_manager.cc:1905] T 3346839176924292b8ff50cbe7bdc15f P 7b178dcd80ae46949a1793f90f4a5cf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:32:08.729347 8535 ts_tablet_manager.cc:1918] T 3346839176924292b8ff50cbe7bdc15f P b2b9ca83586547e18d54d7eb8eed7b76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:32:08.729496 8535 log.cc:1196] T 3346839176924292b8ff50cbe7bdc15f P b2b9ca83586547e18d54d7eb8eed7b76: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-0/wal/wals/3346839176924292b8ff50cbe7bdc15f
I20260926 03:32:08.729969 8535 ts_tablet_manager.cc:1939] T 3346839176924292b8ff50cbe7bdc15f P b2b9ca83586547e18d54d7eb8eed7b76: Deleting consensus metadata
I20260926 03:32:08.730930 8537 ts_tablet_manager.cc:1918] T 3346839176924292b8ff50cbe7bdc15f P 7b178dcd80ae46949a1793f90f4a5cf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:32:08.731078 8537 log.cc:1196] T 3346839176924292b8ff50cbe7bdc15f P 7b178dcd80ae46949a1793f90f4a5cf9: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-1/wal/wals/3346839176924292b8ff50cbe7bdc15f
I20260926 03:32:08.731120 7952 catalog_manager.cc:4841] TS b2b9ca83586547e18d54d7eb8eed7b76 (127.0.97.1:46445): tablet 3346839176924292b8ff50cbe7bdc15f (table test-table [id=4e9007da72fe4ea1afe671b3144e1ebf]) successfully deleted
I20260926 03:32:08.731498 8537 ts_tablet_manager.cc:1939] T 3346839176924292b8ff50cbe7bdc15f P 7b178dcd80ae46949a1793f90f4a5cf9: Deleting consensus metadata
I20260926 03:32:08.732618 7951 catalog_manager.cc:4841] TS 7b178dcd80ae46949a1793f90f4a5cf9 (127.0.97.2:35309): tablet 3346839176924292b8ff50cbe7bdc15f (table test-table [id=4e9007da72fe4ea1afe671b3144e1ebf]) successfully deleted
I20260926 03:32:08.733943 8536 ts_tablet_manager.cc:1918] T 3346839176924292b8ff50cbe7bdc15f P 261504269b4c4c7e9d96e8fd93cff7dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:32:08.734092 8536 log.cc:1196] T 3346839176924292b8ff50cbe7bdc15f P 261504269b4c4c7e9d96e8fd93cff7dc: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790393463219689-388-0/minicluster-data/ts-2/wal/wals/3346839176924292b8ff50cbe7bdc15f
I20260926 03:32:08.734447 8536 ts_tablet_manager.cc:1939] T 3346839176924292b8ff50cbe7bdc15f P 261504269b4c4c7e9d96e8fd93cff7dc: Deleting consensus metadata
I20260926 03:32:08.735385 7951 catalog_manager.cc:4841] TS 261504269b4c4c7e9d96e8fd93cff7dc (127.0.97.3:43645): tablet 3346839176924292b8ff50cbe7bdc15f (table test-table [id=4e9007da72fe4ea1afe671b3144e1ebf]) successfully deleted
Sep 26 03:32:08 dist-test-slave-dt7p krb5kdc[7920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260926 03:32:08.749131 8576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:08.725187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60350 (local address 127.0.97.1:46445)
0926 03:32:08.735552 (+ 10365us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:08.735559 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:08.735974 (+ 415us) server_negotiation.cc:407] Connection header received
0926 03:32:08.736103 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:08.736106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:08.736166 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:08.736254 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:08.737461 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:08.738187 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:08.739378 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:08.739753 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:08.739847 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:08.743719 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:08.743731 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:08.743755 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:08.743790 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:08.746614 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:08.747218 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:08.747225 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:08.747227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:08.747335 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:08.747624 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:08.747627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:08.747629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:08.748007 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:08.748216 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:08.748524 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 03:32:08.748758 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10201,"thread_start_us":10090,"threads_started":1}
I20260926 03:32:08.749745 8167 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:32:08.847352 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 8036
I20260926 03:32:08.866160 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 8176
Sep 26 03:32:08 dist-test-slave-dt7p krb5kdc[7920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260926 03:32:08.881664 8568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:08.867017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54702 (local address 127.0.97.2:35309)
0926 03:32:08.867187 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:08.867192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:08.867210 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:32:08.867335 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:08.867339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:08.867399 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:08.867479 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:08.868717 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:08.869413 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:08.870679 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:08.871298 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:08.871456 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:08.875743 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:08.875758 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:08.875776 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:08.875811 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:08.878896 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:08.879593 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:08.879601 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:08.879603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:08.879730 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:08.880117 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:08.880137 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:08.880140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:08.880492 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:08.880871 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:08.881103 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 03:32:08.881359 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 03:32:08.882361 8306 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:32:09.025918 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 8176
I20260926 03:32:09.052138 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 8315
Sep 26 03:32:09 dist-test-slave-dt7p krb5kdc[7920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260926 03:32:09.067447 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:09.052886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54388 (local address 127.0.97.3:43645)
0926 03:32:09.053180 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:09.053188 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:09.053274 (+ 86us) server_negotiation.cc:407] Connection header received
0926 03:32:09.053482 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:09.053492 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:09.053574 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:09.053676 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:09.054906 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:09.055632 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:09.056899 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:09.057306 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:09.057435 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:09.061733 (+ 4298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:09.061748 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:09.061765 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:09.061799 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:09.064506 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:09.065443 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:09.065463 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:09.065465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:09.065604 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:09.065944 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:09.065950 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:09.065953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:09.066291 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:09.066612 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:09.066881 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 03:32:09.067107 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260926 03:32:09.068095 8445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:32:09.192001 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 8315
I20260926 03:32:09.213855 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 7933
I20260926 03:32:09.215197 8005 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:32:09.328615 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 7933
2026-09-26T03:32:09Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5964 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:32:09 dist-test-slave-dt7p krb5kdc[8583](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:32:09 dist-test-slave-dt7p krb5kdc[8583](info): set up 2 sockets
Sep 26 03:32:09 dist-test-slave-dt7p krb5kdc[8583](info): commencing operation
krb5kdc: starting...
W20260926 03:32:10.477682 388 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:32:10 dist-test-slave-dt7p krb5kdc[8583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:32:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:32:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:32:10.693979 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46779
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:32:11.014526 8596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:32:11.014874 8596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:32:11.015043 8596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:32:11.022217 8596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:32:11.022370 8596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:32:11.022482 8596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:32:11.022574 8596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:32:11.022667 8596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:32:11.032640 8596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:32:11.034041 8596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:32:11.035320 8596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:32:11.046897 8601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:11.046865 8602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:11.046823 8604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:32:11.048058 8596 server_base.cc:1034] running on GCE node
I20260926 03:32:11.048967 8596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:32:11.050310 8596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:32:11.051540 8596 hybrid_clock.cc:648] HybridClock initialized: now 1790393531051500 us; error 62 us; skew 500 ppm
I20260926 03:32:11.051864 8596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T03:32:11Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1790 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:32:11 dist-test-slave-dt7p krb5kdc[8610](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:32:11 dist-test-slave-dt7p krb5kdc[8610](info): set up 2 sockets
Sep 26 03:32:11 dist-test-slave-dt7p krb5kdc[8610](info): commencing operation
krb5kdc: starting...
W20260926 03:32:12.221637 388 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:32:12 dist-test-slave-dt7p krb5kdc[8610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:32:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:32:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:32:12.463791 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38695
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38695
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:32:12.779484 8623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:32:12.779752 8623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:32:12.779922 8623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:32:12.786931 8623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:32:12.787036 8623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:32:12.787170 8623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:32:12.787266 8623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:32:12.787348 8623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:32:12.796967 8623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:32:12.798237 8623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:32:12.799477 8623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:32:12.810519 8628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:12.810566 8629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:12.810598 8631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:32:12.811384 8623 server_base.cc:1034] running on GCE node
I20260926 03:32:12.812090 8623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:32:12.813477 8623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:32:12.814710 8623 hybrid_clock.cc:648] HybridClock initialized: now 1790393532814661 us; error 30 us; skew 500 ppm
I20260926 03:32:12.815088 8623 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:32:12 dist-test-slave-dt7p krb5kdc[8610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:32:12.829483 8623 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:32:12.830911 8623 webserver.cc:460] Webserver started at http://127.0.97.62:39507/ using document root <none> and password file <none>
I20260926 03:32:12.831594 8623 fs_manager.cc:359] Metadata directory not provided
I20260926 03:32:12.831737 8623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:32:12.832050 8623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:32:12.835055 8623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "32be6c88fa574f089e548b5e90418e84"
format_stamp: "Formatted at 2026-09-26 03:32:12 on dist-test-slave-dt7p"
server_key: "750d1a0456ec4f35360485f31fe00c80"
server_key_iv: "0f63aa6264fb4fb3ba40a78d3d18721f"
server_key_version: "encryptionkey@0"
I20260926 03:32:12.835835 8623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "32be6c88fa574f089e548b5e90418e84"
format_stamp: "Formatted at 2026-09-26 03:32:12 on dist-test-slave-dt7p"
server_key: "750d1a0456ec4f35360485f31fe00c80"
server_key_iv: "0f63aa6264fb4fb3ba40a78d3d18721f"
server_key_version: "encryptionkey@0"
I20260926 03:32:12.841198 8623 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260926 03:32:12.844632 8638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:12.845916 8623 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:32:12.846130 8623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "32be6c88fa574f089e548b5e90418e84"
format_stamp: "Formatted at 2026-09-26 03:32:12 on dist-test-slave-dt7p"
server_key: "750d1a0456ec4f35360485f31fe00c80"
server_key_iv: "0f63aa6264fb4fb3ba40a78d3d18721f"
server_key_version: "encryptionkey@0"
I20260926 03:32:12.846326 8623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:32:12.859911 8623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:32:12.864068 8623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:32:12.864334 8623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:32:12.882211 8623 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38695
I20260926 03:32:12.882234 8689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38695 every 8 connection(s)
I20260926 03:32:12.883492 8623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:32:12.884140 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 8623
I20260926 03:32:12.884377 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:32:12.885095 388 external_mini_cluster.cc:1428] Setting key 5f27302e7cc6651f1c2eafd935ca26aa
I20260926 03:32:12.887626 8690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:32:12.897549 8690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84: Bootstrap starting.
Sep 26 03:32:12 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:32:12.901036 8690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84: Neither blocks nor log segments found. Creating new log.
I20260926 03:32:12.902037 8690 log.cc:824] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84: Log is configured to *not* fsync() on all Append() calls
I20260926 03:32:12.905174 8690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84: No bootstrap required, opened a new log
I20260926 03:32:12.905972 8693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:12.886971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34708 (local address 127.0.97.62:38695)
0926 03:32:12.887682 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:12.887693 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:12.887753 (+ 60us) server_negotiation.cc:407] Connection header received
0926 03:32:12.889376 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:12.889395 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:12.889728 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:12.890101 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:12.891471 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:12.892801 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:12.893984 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:12.894621 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:12.894820 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:12.898914 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:12.898921 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:12.898933 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:12.898968 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:12.901989 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:12.902821 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:12.902828 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:12.902836 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:12.903071 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:12.903433 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:12.903436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:12.903438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:12.903894 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:12.904094 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:12.904620 (+ 526us) server_negotiation.cc:299] Negotiation successful
0926 03:32:12.904985 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":329,"threads_started":1}
I20260926 03:32:12.912928 8690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32be6c88fa574f089e548b5e90418e84" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38695 } }
I20260926 03:32:12.913270 8690 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:32:12.913375 8690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32be6c88fa574f089e548b5e90418e84, State: Initialized, Role: FOLLOWER
I20260926 03:32:12.913895 8690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32be6c88fa574f089e548b5e90418e84" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38695 } }
I20260926 03:32:12.914122 8690 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:32:12.914273 8690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:32:12.914458 8690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:32:12.916754 8690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32be6c88fa574f089e548b5e90418e84" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38695 } }
I20260926 03:32:12.917227 8690 leader_election.cc:302] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32be6c88fa574f089e548b5e90418e84; no voters:
I20260926 03:32:12.917757 8690 leader_election.cc:288] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:32:12.917886 8695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:32:12.918836 8695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84 [term 1 LEADER]: Becoming Leader. State: Replica: 32be6c88fa574f089e548b5e90418e84, State: Running, Role: LEADER
I20260926 03:32:12.919453 8695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32be6c88fa574f089e548b5e90418e84" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38695 } }
I20260926 03:32:12.919935 8690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:32:12.922457 8697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32be6c88fa574f089e548b5e90418e84. Latest consensus state: current_term: 1 leader_uuid: "32be6c88fa574f089e548b5e90418e84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32be6c88fa574f089e548b5e90418e84" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38695 } } }
I20260926 03:32:12.922485 8696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32be6c88fa574f089e548b5e90418e84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32be6c88fa574f089e548b5e90418e84" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38695 } } }
I20260926 03:32:12.922997 8696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84 [sys.catalog]: This master's current role is: LEADER
I20260926 03:32:12.922993 8697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84 [sys.catalog]: This master's current role is: LEADER
I20260926 03:32:12.924505 8701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:32:12.929522 8701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:32:12.938119 8701 catalog_manager.cc:1347] Generated new cluster ID: e2c31ecb27614434a241ad6fad5a67af
I20260926 03:32:12.938354 8701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:32:12.945883 8701 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:32:12.947149 8701 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:32:12.958249 8701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84: Generated new TSK 0
I20260926 03:32:12.959038 8701 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:32:13.014570 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38695
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:32:13.333164 8716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:32:13.333485 8716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:32:13.333581 8716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:32:13.340467 8716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:32:13.340620 8716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:32:13.340752 8716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:32:13.351176 8716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38695
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:32:13.352408 8716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:32:13.353897 8716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:32:13.366186 8721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:13.366137 8722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:13.366271 8724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:32:13.369984 8716 server_base.cc:1034] running on GCE node
I20260926 03:32:13.370571 8716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:32:13.371529 8716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:32:13.372747 8716 hybrid_clock.cc:648] HybridClock initialized: now 1790393533372712 us; error 45 us; skew 500 ppm
I20260926 03:32:13.373080 8716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:32:13 dist-test-slave-dt7p krb5kdc[8610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:32:13.376152 8716 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:32:13.377612 8716 webserver.cc:460] Webserver started at http://127.0.97.1:36885/ using document root <none> and password file <none>
I20260926 03:32:13.378311 8716 fs_manager.cc:359] Metadata directory not provided
I20260926 03:32:13.378453 8716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:32:13.378757 8716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:32:13.381608 8716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-0/data/instance:
uuid: "e37ddc437b1a405ea3fc9807221afc9c"
format_stamp: "Formatted at 2026-09-26 03:32:13 on dist-test-slave-dt7p"
server_key: "a1adaca28f243aedf0467f62cc8c9456"
server_key_iv: "68d530705a403675d953a2fc2b26080b"
server_key_version: "encryptionkey@0"
I20260926 03:32:13.382378 8716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e37ddc437b1a405ea3fc9807221afc9c"
format_stamp: "Formatted at 2026-09-26 03:32:13 on dist-test-slave-dt7p"
server_key: "a1adaca28f243aedf0467f62cc8c9456"
server_key_iv: "68d530705a403675d953a2fc2b26080b"
server_key_version: "encryptionkey@0"
I20260926 03:32:13.387394 8716 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 03:32:13.390618 8731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:13.391894 8716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 03:32:13.392112 8716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-0/wal
uuid: "e37ddc437b1a405ea3fc9807221afc9c"
format_stamp: "Formatted at 2026-09-26 03:32:13 on dist-test-slave-dt7p"
server_key: "a1adaca28f243aedf0467f62cc8c9456"
server_key_iv: "68d530705a403675d953a2fc2b26080b"
server_key_version: "encryptionkey@0"
I20260926 03:32:13.392329 8716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:32:13.408252 8716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:32:13.412029 8716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:32:13.412310 8716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:32:13.413286 8716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:32:13.414562 8716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:32:13.414669 8716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:13.414811 8716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:32:13.414861 8716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:13.447532 8716 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40327
I20260926 03:32:13.447609 8843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40327 every 8 connection(s)
I20260926 03:32:13.448849 8716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:32:13.455749 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 8716
I20260926 03:32:13.455956 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:32:13.456542 388 external_mini_cluster.cc:1428] Setting key 8b878688a50e10c7da6c5548e6a6be7c
Sep 26 03:32:13 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:32:13.469980 8846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:13.451355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59617 (local address 127.0.97.62:38695)
0926 03:32:13.451777 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:13.451781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:13.452768 (+ 987us) server_negotiation.cc:407] Connection header received
0926 03:32:13.454630 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:13.454636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:13.454721 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:13.454834 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:13.456340 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:13.457208 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:13.459216 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:13.459602 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:13.459694 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:13.463848 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:13.463860 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:13.463879 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:13.463918 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:13.466978 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:13.467636 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:13.467644 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:13.467651 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:13.467796 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:13.468163 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:13.468170 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:13.468176 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:13.468535 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:13.468702 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:13.469279 (+ 577us) server_negotiation.cc:299] Negotiation successful
0926 03:32:13.469513 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":220,"threads_started":1}
I20260926 03:32:13.470556 8847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:13.451912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.1:59617)
0926 03:32:13.452625 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:13.452667 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:13.454358 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:13.455055 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:13.455062 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:13.455572 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:13.456185 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:13.456203 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:13.457384 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:13.457392 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:13.459076 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:13.459091 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:13.459814 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:13.459824 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:13.460154 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:13.460906 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:13.460937 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:13.463562 (+ 2625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:32:13.467110 (+ 3548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:32:13.467116 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:32:13.467127 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:32:13.467525 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:32:13.467895 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:32:13.467898 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:32:13.467900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:32:13.468063 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:32:13.468668 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:32:13.468673 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:32:13.468949 (+ 276us) client_negotiation.cc:769] Sending connection context
0926 03:32:13.469165 (+ 216us) client_negotiation.cc:240] Negotiation successful
0926 03:32:13.469512 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":249,"threads_started":1}
I20260926 03:32:13.472098 8844 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38695
I20260926 03:32:13.472409 8844 heartbeater.cc:464] Registering TS with master...
I20260926 03:32:13.473260 8844 heartbeater.cc:511] Master 127.0.97.62:38695 requested a full tablet report, sending...
I20260926 03:32:13.475350 8655 ts_manager.cc:194] Registered new tserver with Master: e37ddc437b1a405ea3fc9807221afc9c (127.0.97.1:40327)
I20260926 03:32:13.476795 8655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59617
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:32:13.505980 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38695
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:32:13.818041 8850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:32:13.818353 8850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:32:13.818526 8850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:32:13.827175 8850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:32:13.827315 8850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:32:13.827481 8850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:32:13.837031 8850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38695
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:32:13.838301 8850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:32:13.839574 8850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:32:13.852427 8858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:13.852516 8856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:13.852663 8855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:32:13.853713 8850 server_base.cc:1034] running on GCE node
I20260926 03:32:13.854326 8850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:32:13.855197 8850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:32:13.856429 8850 hybrid_clock.cc:648] HybridClock initialized: now 1790393533856384 us; error 49 us; skew 500 ppm
I20260926 03:32:13.856813 8850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:32:13 dist-test-slave-dt7p krb5kdc[8610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:32:13.859754 8850 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:32:13.861228 8850 webserver.cc:460] Webserver started at http://127.0.97.2:33293/ using document root <none> and password file <none>
I20260926 03:32:13.861831 8850 fs_manager.cc:359] Metadata directory not provided
I20260926 03:32:13.861929 8850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:32:13.862175 8850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:32:13.864971 8850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-1/data/instance:
uuid: "7c201ec753f341fca08f30091bf358a6"
format_stamp: "Formatted at 2026-09-26 03:32:13 on dist-test-slave-dt7p"
server_key: "ac3a50b29bb7f8f1193317cedb068c21"
server_key_iv: "501bee307357a7c506357208e02759cd"
server_key_version: "encryptionkey@0"
I20260926 03:32:13.865715 8850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7c201ec753f341fca08f30091bf358a6"
format_stamp: "Formatted at 2026-09-26 03:32:13 on dist-test-slave-dt7p"
server_key: "ac3a50b29bb7f8f1193317cedb068c21"
server_key_iv: "501bee307357a7c506357208e02759cd"
server_key_version: "encryptionkey@0"
I20260926 03:32:13.870648 8850 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:32:13.873957 8865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:13.875352 8850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:32:13.875538 8850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-1/wal
uuid: "7c201ec753f341fca08f30091bf358a6"
format_stamp: "Formatted at 2026-09-26 03:32:13 on dist-test-slave-dt7p"
server_key: "ac3a50b29bb7f8f1193317cedb068c21"
server_key_iv: "501bee307357a7c506357208e02759cd"
server_key_version: "encryptionkey@0"
I20260926 03:32:13.875746 8850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:32:13.896775 8850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:32:13.902524 8850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:32:13.902855 8850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:32:13.904036 8850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:32:13.906533 8850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:32:13.906627 8850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:13.906759 8850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:32:13.906821 8850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:13.938241 8850 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42163
I20260926 03:32:13.938293 8977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42163 every 8 connection(s)
I20260926 03:32:13.939495 8850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:32:13.945217 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 8850
I20260926 03:32:13.945488 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:32:13.946103 388 external_mini_cluster.cc:1428] Setting key 86107a98b19dd2db33193de4f12ca60b
Sep 26 03:32:13 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:32:13.962889 8846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:13.942022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60357 (local address 127.0.97.62:38695)
0926 03:32:13.942192 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:13.942196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:13.943303 (+ 1107us) server_negotiation.cc:407] Connection header received
0926 03:32:13.945194 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:13.945198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:13.945263 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:13.945390 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:13.947373 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:13.948219 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:13.950755 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:13.951221 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:13.951322 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:13.955880 (+ 4558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:13.955888 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:13.955900 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:13.955993 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:13.959621 (+ 3628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:13.960289 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:13.960297 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:13.960304 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:13.960465 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:13.960905 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:13.960920 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:13.960926 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:13.961273 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:13.961421 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:13.962228 (+ 807us) server_negotiation.cc:299] Negotiation successful
0926 03:32:13.962461 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 03:32:13.963508 8980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:13.942419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.2:60357)
0926 03:32:13.943149 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:13.943188 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:13.944854 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:13.945650 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:13.945662 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:13.946570 (+ 908us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:13.947196 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:13.947218 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:13.948365 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:13.948371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:13.950585 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:13.950603 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:13.951454 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:13.951463 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:13.951776 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:13.952489 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:13.952528 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:13.955495 (+ 2967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:32:13.959753 (+ 4258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:32:13.959761 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:32:13.959771 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:32:13.960164 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:32:13.960622 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:32:13.960625 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:32:13.960628 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:32:13.960782 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:32:13.961384 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:32:13.961389 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:32:13.961831 (+ 442us) client_negotiation.cc:769] Sending connection context
0926 03:32:13.962081 (+ 250us) client_negotiation.cc:240] Negotiation successful
0926 03:32:13.962440 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":277,"threads_started":1}
I20260926 03:32:13.965044 8978 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38695
I20260926 03:32:13.965335 8978 heartbeater.cc:464] Registering TS with master...
I20260926 03:32:13.966095 8978 heartbeater.cc:511] Master 127.0.97.62:38695 requested a full tablet report, sending...
I20260926 03:32:13.967525 8655 ts_manager.cc:194] Registered new tserver with Master: 7c201ec753f341fca08f30091bf358a6 (127.0.97.2:42163)
I20260926 03:32:13.968343 8655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:60357
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:32:13.997805 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38695
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:32:14.314888 8983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:32:14.315198 8983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:32:14.315371 8983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:32:14.322499 8983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:32:14.322639 8983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:32:14.322813 8983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:32:14.332281 8983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38695
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:32:14.333556 8983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:32:14.334824 8983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:32:14.348148 8988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:14.348318 8991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:14.348169 8989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:32:14.348943 8983 server_base.cc:1034] running on GCE node
I20260926 03:32:14.349524 8983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:32:14.350574 8983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:32:14.351827 8983 hybrid_clock.cc:648] HybridClock initialized: now 1790393534351774 us; error 59 us; skew 500 ppm
I20260926 03:32:14.352198 8983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:32:14 dist-test-slave-dt7p krb5kdc[8610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:32:14.355523 8983 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:32:14.357082 8983 webserver.cc:460] Webserver started at http://127.0.97.3:37281/ using document root <none> and password file <none>
I20260926 03:32:14.357779 8983 fs_manager.cc:359] Metadata directory not provided
I20260926 03:32:14.357913 8983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:32:14.358223 8983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:32:14.361143 8983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-2/data/instance:
uuid: "cf30087daece4deab040144cc0f164ed"
format_stamp: "Formatted at 2026-09-26 03:32:14 on dist-test-slave-dt7p"
server_key: "59f52e18b2deb5d005afc883ec990846"
server_key_iv: "6c2c98d5fab81ebc08cc6051d1034d50"
server_key_version: "encryptionkey@0"
I20260926 03:32:14.361881 8983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-2/wal/instance:
uuid: "cf30087daece4deab040144cc0f164ed"
format_stamp: "Formatted at 2026-09-26 03:32:14 on dist-test-slave-dt7p"
server_key: "59f52e18b2deb5d005afc883ec990846"
server_key_iv: "6c2c98d5fab81ebc08cc6051d1034d50"
server_key_version: "encryptionkey@0"
I20260926 03:32:14.367100 8983 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:32:14.370815 8998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:14.372234 8983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:32:14.372417 8983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-2/wal
uuid: "cf30087daece4deab040144cc0f164ed"
format_stamp: "Formatted at 2026-09-26 03:32:14 on dist-test-slave-dt7p"
server_key: "59f52e18b2deb5d005afc883ec990846"
server_key_iv: "6c2c98d5fab81ebc08cc6051d1034d50"
server_key_version: "encryptionkey@0"
I20260926 03:32:14.372649 8983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:32:14.384824 8983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:32:14.388831 8983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:32:14.389122 8983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:32:14.390023 8983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:32:14.391302 8983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:32:14.391402 8983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:14.391541 8983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:32:14.391610 8983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:14.423739 8983 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33551
I20260926 03:32:14.423790 9110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33551 every 8 connection(s)
I20260926 03:32:14.425124 8983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:32:14.428707 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 8983
I20260926 03:32:14.428923 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:32:14.429448 388 external_mini_cluster.cc:1428] Setting key 73df043298f49ffa2f85e2a9c6b3226c
Sep 26 03:32:14 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:32:14.445590 8846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:14.427743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44933 (local address 127.0.97.62:38695)
0926 03:32:14.427977 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:14.427984 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:14.429349 (+ 1365us) server_negotiation.cc:407] Connection header received
0926 03:32:14.431637 (+ 2288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:14.431644 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:14.431722 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:14.431811 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:14.433564 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:14.434317 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:14.435796 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:14.436197 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:14.436300 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:14.439608 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:14.439613 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:14.439616 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:14.439659 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:14.442327 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:14.443070 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:14.443074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:14.443076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:14.443201 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:14.443618 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:14.443628 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:14.443633 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:14.444003 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:14.444165 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:14.445040 (+ 875us) server_negotiation.cc:299] Negotiation successful
0926 03:32:14.445258 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 03:32:14.446517 9113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:14.428301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.3:44933)
0926 03:32:14.429168 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:14.429220 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:14.431395 (+ 2175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:14.432069 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:14.432081 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:14.432840 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:14.433420 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:14.433437 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:14.434435 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:14.434438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:14.435639 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:14.435654 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:14.436379 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:14.436384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:14.436638 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:14.437283 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:14.437304 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:14.439400 (+ 2096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:32:14.442466 (+ 3066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:32:14.442474 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:32:14.442488 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:32:14.442943 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:32:14.443309 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:32:14.443315 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:32:14.443322 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:32:14.443506 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:32:14.444140 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:32:14.444147 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:32:14.444550 (+ 403us) client_negotiation.cc:769] Sending connection context
0926 03:32:14.444909 (+ 359us) client_negotiation.cc:240] Negotiation successful
0926 03:32:14.445448 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":353,"threads_started":1}
I20260926 03:32:14.447911 9111 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38695
I20260926 03:32:14.448213 9111 heartbeater.cc:464] Registering TS with master...
I20260926 03:32:14.448936 9111 heartbeater.cc:511] Master 127.0.97.62:38695 requested a full tablet report, sending...
I20260926 03:32:14.450459 8655 ts_manager.cc:194] Registered new tserver with Master: cf30087daece4deab040144cc0f164ed (127.0.97.3:33551)
I20260926 03:32:14.451545 8655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44933
I20260926 03:32:14.457294 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
I20260926 03:32:14.480077 8844 heartbeater.cc:503] Master 127.0.97.62:38695 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260926 03:32:14.596995 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 8716
Sep 26 03:32:14 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:32:14.612703 9121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:14.598073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45580 (local address 127.0.97.1:40327)
0926 03:32:14.598704 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:14.598712 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:14.598739 (+ 27us) server_negotiation.cc:407] Connection header received
0926 03:32:14.598803 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:14.598809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:14.599034 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:14.599176 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:14.600434 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:14.601561 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:14.602736 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:14.603294 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:14.603395 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:14.607215 (+ 3820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:14.607226 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:14.607241 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:14.607278 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:14.610106 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:14.610778 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:14.610785 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:14.610788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:14.610922 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:14.611286 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:14.611289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:14.611291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:14.611639 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:14.611854 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:14.612135 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 03:32:14.612324 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":305,"threads_started":1}
I20260926 03:32:14.613946 8839 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:32:14.697324 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 8716
I20260926 03:32:14.713703 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 8850
Sep 26 03:32:14 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:32:14.730075 9123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:14.714575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45804 (local address 127.0.97.2:42163)
0926 03:32:14.715150 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:14.715157 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:14.715183 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:32:14.715241 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:14.715247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:14.715443 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:14.715579 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:14.716740 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:14.717735 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:14.718980 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:14.719502 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:14.719603 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:14.723850 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:14.723869 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:14.723889 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:14.723945 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:14.727168 (+ 3223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:14.727779 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:14.727786 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:14.727789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:14.727922 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:14.728228 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:14.728232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:14.728234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:14.728884 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:14.729134 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:14.729424 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 03:32:14.729683 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":302,"threads_started":1}
I20260926 03:32:14.731385 8973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:32:14.815591 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 8850
I20260926 03:32:14.832819 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 8983
Sep 26 03:32:14 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:32:14.849251 9125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:14.833896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52378 (local address 127.0.97.3:33551)
0926 03:32:14.834502 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:14.834510 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:14.834539 (+ 29us) server_negotiation.cc:407] Connection header received
0926 03:32:14.834597 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:14.834603 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:14.834818 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:14.834941 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:14.836144 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:14.837364 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:14.838645 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:14.839189 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:14.839290 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:14.843767 (+ 4477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:14.843780 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:14.843794 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:14.843829 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:14.846578 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:14.847235 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:14.847242 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:14.847244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:14.847361 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:14.847700 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:14.847703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:14.847705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:14.848047 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:14.848277 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:14.848551 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 03:32:14.848911 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":317,"threads_started":1}
I20260926 03:32:14.850620 9106 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:32:14.934234 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 8983
I20260926 03:32:14.951825 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 8623
I20260926 03:32:14.953008 8685 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:32:15.056294 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 8623
I20260926 03:32:15.076854 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38695
--webserver_interface=127.0.97.62
--webserver_port=39507
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38695
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:32:15.398293 9128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:32:15.398737 9128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:32:15.398914 9128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:32:15.406152 9128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:32:15.406289 9128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:32:15.406397 9128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:32:15.406517 9128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:32:15.406594 9128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:32:15.416227 9128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=39507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:32:15.417604 9128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:32:15.418874 9128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:32:15.431633 9133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:15.431710 9134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:15.431666 9136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:32:15.433005 9128 server_base.cc:1034] running on GCE node
I20260926 03:32:15.433837 9128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:32:15.435331 9128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:32:15.436641 9128 hybrid_clock.cc:648] HybridClock initialized: now 1790393535436529 us; error 68 us; skew 500 ppm
I20260926 03:32:15.437004 9128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:32:15 dist-test-slave-dt7p krb5kdc[8610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:32:15.440181 9128 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:32:15.441797 9128 webserver.cc:460] Webserver started at http://127.0.97.62:39507/ using document root <none> and password file <none>
I20260926 03:32:15.442546 9128 fs_manager.cc:359] Metadata directory not provided
I20260926 03:32:15.442709 9128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:32:15.448205 9128 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20260926 03:32:15.450945 9143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:15.452378 9128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 03:32:15.452629 9128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "32be6c88fa574f089e548b5e90418e84"
format_stamp: "Formatted at 2026-09-26 03:32:12 on dist-test-slave-dt7p"
server_key: "750d1a0456ec4f35360485f31fe00c80"
server_key_iv: "0f63aa6264fb4fb3ba40a78d3d18721f"
server_key_version: "encryptionkey@0"
I20260926 03:32:15.453580 9128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:32:15.482317 9128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:32:15.486541 9128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:32:15.486856 9128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:32:15.505056 9128 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38695
I20260926 03:32:15.505084 9194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38695 every 8 connection(s)
I20260926 03:32:15.506505 9128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:32:15.508790 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 9128
I20260926 03:32:15.509517 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:40327
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:38695
--webserver_port=36885
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:32:15.513185 9195 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:32:15.516239 9195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84: Bootstrap starting.
I20260926 03:32:15.535383 9195 log.cc:824] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84: Log is configured to *not* fsync() on all Append() calls
I20260926 03:32:15.543556 9195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:32:15.544111 9195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84: Bootstrap complete.
I20260926 03:32:15.552879 9195 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32be6c88fa574f089e548b5e90418e84" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38695 } }
I20260926 03:32:15.553679 9195 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32be6c88fa574f089e548b5e90418e84, State: Initialized, Role: FOLLOWER
I20260926 03:32:15.554224 9195 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32be6c88fa574f089e548b5e90418e84" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38695 } }
I20260926 03:32:15.554454 9195 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:32:15.554577 9195 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:32:15.554746 9195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84 [term 1 FOLLOWER]: Advancing to term 2
I20260926 03:32:15.558543 9195 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32be6c88fa574f089e548b5e90418e84" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38695 } }
I20260926 03:32:15.558997 9195 leader_election.cc:302] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32be6c88fa574f089e548b5e90418e84; no voters:
I20260926 03:32:15.559614 9195 leader_election.cc:288] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 03:32:15.559748 9199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 03:32:15.561885 9195 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:32:15.564960 9199 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84 [term 2 LEADER]: Becoming Leader. State: Replica: 32be6c88fa574f089e548b5e90418e84, State: Running, Role: LEADER
I20260926 03:32:15.565683 9199 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32be6c88fa574f089e548b5e90418e84" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38695 } }
I20260926 03:32:15.568606 9201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32be6c88fa574f089e548b5e90418e84. Latest consensus state: current_term: 2 leader_uuid: "32be6c88fa574f089e548b5e90418e84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32be6c88fa574f089e548b5e90418e84" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38695 } } }
I20260926 03:32:15.568601 9200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "32be6c88fa574f089e548b5e90418e84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32be6c88fa574f089e548b5e90418e84" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38695 } } }
I20260926 03:32:15.569032 9201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84 [sys.catalog]: This master's current role is: LEADER
I20260926 03:32:15.569038 9200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84 [sys.catalog]: This master's current role is: LEADER
I20260926 03:32:15.576164 9204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:32:15.582845 9204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:32:15.584649 9204 catalog_manager.cc:1259] Loaded cluster ID: e2c31ecb27614434a241ad6fad5a67af
I20260926 03:32:15.584807 9204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:32:15.589782 9204 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:32:15.591612 9204 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 32be6c88fa574f089e548b5e90418e84: Loaded TSK: 0
I20260926 03:32:15.593441 9204 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 03:32:15.867533 9197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:32:15.867807 9197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:32:15.867959 9197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:32:15.875176 9197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:32:15.875288 9197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:32:15.875466 9197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:32:15.885349 9197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:40327
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=36885
--tserver_master_addrs=127.0.97.62:38695
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:32:15.886688 9197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:32:15.887938 9197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:32:15.902622 9222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:15.904973 9224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:15.905997 9221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:32:15.906644 9197 server_base.cc:1034] running on GCE node
I20260926 03:32:15.907617 9197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:32:15.908951 9197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:32:15.910316 9197 hybrid_clock.cc:648] HybridClock initialized: now 1790393535910239 us; error 62 us; skew 500 ppm
I20260926 03:32:15.910732 9197 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:32:15 dist-test-slave-dt7p krb5kdc[8610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393535, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:32:15.914013 9197 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260926 03:32:15.915707 9197 webserver.cc:460] Webserver started at http://127.0.97.1:36885/ using document root <none> and password file <none>
I20260926 03:32:15.916376 9197 fs_manager.cc:359] Metadata directory not provided
I20260926 03:32:15.916512 9197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:32:15.921809 9197 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:32:15.924444 9231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:15.925834 9197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:32:15.926005 9197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-0/wal
uuid: "e37ddc437b1a405ea3fc9807221afc9c"
format_stamp: "Formatted at 2026-09-26 03:32:13 on dist-test-slave-dt7p"
server_key: "a1adaca28f243aedf0467f62cc8c9456"
server_key_iv: "68d530705a403675d953a2fc2b26080b"
server_key_version: "encryptionkey@0"
I20260926 03:32:15.926898 9197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:32:15.949604 9197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:32:15.953945 9197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:32:15.954216 9197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:32:15.955253 9197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:32:15.956634 9197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:32:15.956718 9197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:15.956861 9197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:32:15.956979 9197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:15.989588 9197 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40327
I20260926 03:32:15.989678 9343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40327 every 8 connection(s)
I20260926 03:32:15.991211 9197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:32:16.001698 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 9197
I20260926 03:32:16.002285 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:42163
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:38695
--webserver_port=33293
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:32:16 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:16 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 03:32:16.009723 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:15.994306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54865 (local address 127.0.97.62:38695)
0926 03:32:15.995009 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:15.995019 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:15.995623 (+ 604us) server_negotiation.cc:407] Connection header received
0926 03:32:15.997653 (+ 2030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:15.997665 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:15.997959 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:15.998266 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:15.999638 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:16.000916 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:16.002572 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:16.003203 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:16.003411 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:16.008885 (+ 5474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54865: BlockingRecv error: recv got EOF from 127.0.97.1:54865 (error 108)
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":291,"threads_started":1}
I20260926 03:32:16.009791 9347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:15.994320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.1:54865)
0926 03:32:15.995446 (+ 1126us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:15.995483 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:15.997180 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:15.998405 (+ 1225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:15.998413 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:15.998937 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:15.999458 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:15.999474 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:16.001055 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:16.001059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:16.002368 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:16.002387 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:16.003478 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:16.003486 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:16.003735 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:16.004414 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:16.004441 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:16.008425 (+ 3984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":909,"thread_start_us":708,"threads_started":1}
W20260926 03:32:16.010560 9344 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38695 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:38695: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 26 03:32:16 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:16 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:16.019616 9347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:16.011186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.1:50119)
0926 03:32:16.011367 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:16.011386 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:16.011494 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:16.011902 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:16.011906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:16.012173 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:16.012303 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:16.012310 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:16.013998 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:16.014002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:16.014891 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:16.014901 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:16.016010 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:16.016014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:16.016131 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:16.016776 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:16.016792 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:16.019417 (+ 2625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 03:32:16.020047 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:16.011385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50119 (local address 127.0.97.62:38695)
0926 03:32:16.011590 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:16.011599 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:16.011620 (+ 21us) server_negotiation.cc:407] Connection header received
0926 03:32:16.011680 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:16.011688 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:16.011771 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:16.011881 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:16.012434 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:16.013835 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:16.015029 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:16.015834 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:16.016016 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:16.019736 (+ 3720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:50119: BlockingRecv error: recv got EOF from 127.0.97.1:50119 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 03:32:16 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:16 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:16.028312 9347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:16.020339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.1:41815)
0926 03:32:16.020477 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:16.020494 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:16.020645 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:16.020987 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:16.020991 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:16.021288 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:16.021415 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:16.021423 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:16.022380 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:16.022385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:16.023275 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:16.023285 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:16.023975 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:16.023981 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:16.024160 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:16.025003 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:16.025068 (+ 65us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:16.027983 (+ 2915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 03:32:16.028671 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:16.020517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41815 (local address 127.0.97.62:38695)
0926 03:32:16.020742 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:16.020748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:16.020764 (+ 16us) server_negotiation.cc:407] Connection header received
0926 03:32:16.020805 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:16.020808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:16.020869 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:16.020969 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:16.021556 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:16.022242 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:16.023416 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:16.023831 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:16.023966 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:16.028203 (+ 4237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:41815: BlockingRecv error: recv got EOF from 127.0.97.1:41815 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260926 03:32:16.028764 9344 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 03:32:16.328200 9348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:32:16.328532 9348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:32:16.328723 9348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:32:16.336228 9348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:32:16.336395 9348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:32:16.336639 9348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:32:16.347064 9348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:42163
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=33293
--tserver_master_addrs=127.0.97.62:38695
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:32:16.348483 9348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:32:16.349829 9348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:32:16.363364 9356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:16.363433 9354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:16.363435 9353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:32:16.364405 9348 server_base.cc:1034] running on GCE node
I20260926 03:32:16.365278 9348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:32:16.366398 9348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:32:16.367626 9348 hybrid_clock.cc:648] HybridClock initialized: now 1790393536367563 us; error 58 us; skew 500 ppm
I20260926 03:32:16.367977 9348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:32:16 dist-test-slave-dt7p krb5kdc[8610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393536, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:32:16.371251 9348 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260926 03:32:16.372756 9348 webserver.cc:460] Webserver started at http://127.0.97.2:33293/ using document root <none> and password file <none>
I20260926 03:32:16.373461 9348 fs_manager.cc:359] Metadata directory not provided
I20260926 03:32:16.373603 9348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:32:16.379243 9348 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260926 03:32:16.382095 9363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:16.383504 9348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:32:16.383724 9348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-1/wal
uuid: "7c201ec753f341fca08f30091bf358a6"
format_stamp: "Formatted at 2026-09-26 03:32:13 on dist-test-slave-dt7p"
server_key: "ac3a50b29bb7f8f1193317cedb068c21"
server_key_iv: "501bee307357a7c506357208e02759cd"
server_key_version: "encryptionkey@0"
I20260926 03:32:16.384753 9348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:32:16.407500 9348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:32:16.412043 9348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:32:16.412348 9348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:32:16.413434 9348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:32:16.414870 9348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:32:16.414965 9348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:16.415118 9348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:32:16.415210 9348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:16.450845 9348 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42163
I20260926 03:32:16.450894 9475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42163 every 8 connection(s)
I20260926 03:32:16.452392 9348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:32:16.457088 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 9348
I20260926 03:32:16.457839 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:33551
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:38695
--webserver_port=37281
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:32:16 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:16 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 03:32:16.471503 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:16.455032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50337 (local address 127.0.97.62:38695)
0926 03:32:16.455271 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:16.455278 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:16.456521 (+ 1243us) server_negotiation.cc:407] Connection header received
0926 03:32:16.459301 (+ 2780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:16.459305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:16.459374 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:16.459466 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:16.461564 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:16.462521 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:16.465093 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:16.465568 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:16.465669 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:16.471284 (+ 5615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:50337: BlockingRecv error: recv got EOF from 127.0.97.2:50337 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 03:32:16.472442 9478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:16.455596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.2:50337)
0926 03:32:16.456356 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:16.456402 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:16.458976 (+ 2574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:16.459734 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:16.459749 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:16.460542 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:16.461360 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:16.461386 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:16.462697 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:16.462704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:16.464934 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:16.464952 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:16.465822 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:16.465831 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:16.466145 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:16.466916 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:16.466943 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:16.471097 (+ 4154us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":270,"threads_started":1}
W20260926 03:32:16.473218 9476 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38695 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:38695: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 26 03:32:16 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:16 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 03:32:16.482620 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:16.474024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47237 (local address 127.0.97.62:38695)
0926 03:32:16.474212 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:16.474218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:16.474239 (+ 21us) server_negotiation.cc:407] Connection header received
0926 03:32:16.474302 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:16.474307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:16.474382 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:16.474497 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:16.475247 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:16.476039 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:16.477567 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:16.478013 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:16.478156 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:16.482440 (+ 4284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:47237: BlockingRecv error: recv got EOF from 127.0.97.2:47237 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 03:32:16.482648 9478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:16.473869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.2:47237)
0926 03:32:16.474045 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:16.474088 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:16.474197 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:16.474519 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:16.474524 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:16.474876 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:16.475106 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:16.475115 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:16.476185 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:16.476190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:16.477428 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:16.477440 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:16.478164 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:16.478170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:16.478358 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:16.479006 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:16.479025 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:16.482250 (+ 3225us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
Sep 26 03:32:16 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:16 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:16.491791 9478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:16.483448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.2:33217)
0926 03:32:16.483631 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:16.483652 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:16.483753 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:16.484096 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:16.484101 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:16.484416 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:16.484621 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:16.484638 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:16.485786 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:16.485792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:16.486791 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:16.486802 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:16.487555 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:16.487561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:16.487736 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:16.488377 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:16.488396 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:16.491464 (+ 3068us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 03:32:16.492113 9476 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 03:32:16.491881 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:16.483542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33217 (local address 127.0.97.62:38695)
0926 03:32:16.483748 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:16.483753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:16.483772 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:32:16.483861 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:16.483866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:16.483935 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:16.484084 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:16.484763 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:16.485639 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:16.486942 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:16.487389 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:16.487531 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:16.491654 (+ 4123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:33217: BlockingRecv error: recv got EOF from 127.0.97.2:33217 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260926 03:32:16.783226 9479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:32:16.783528 9479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:32:16.783672 9479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:32:16.790899 9479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:32:16.791069 9479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:32:16.791244 9479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:32:16.800491 9479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:33551
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=37281
--tserver_master_addrs=127.0.97.62:38695
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:32:16.801832 9479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:32:16.803069 9479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:32:16.815323 9484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:16.815277 9487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:16.815315 9485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:32:16.816103 9479 server_base.cc:1034] running on GCE node
I20260926 03:32:16.816802 9479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:32:16.817659 9479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:32:16.818882 9479 hybrid_clock.cc:648] HybridClock initialized: now 1790393536818848 us; error 46 us; skew 500 ppm
I20260926 03:32:16.819177 9479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:32:16 dist-test-slave-dt7p krb5kdc[8610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393536, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:32:16.822139 9479 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260926 03:32:16.823455 9479 webserver.cc:460] Webserver started at http://127.0.97.3:37281/ using document root <none> and password file <none>
I20260926 03:32:16.824112 9479 fs_manager.cc:359] Metadata directory not provided
I20260926 03:32:16.824263 9479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:32:16.829391 9479 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:32:16.832062 9494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:16.833635 9479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 03:32:16.833807 9479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-2/wal
uuid: "cf30087daece4deab040144cc0f164ed"
format_stamp: "Formatted at 2026-09-26 03:32:14 on dist-test-slave-dt7p"
server_key: "59f52e18b2deb5d005afc883ec990846"
server_key_iv: "6c2c98d5fab81ebc08cc6051d1034d50"
server_key_version: "encryptionkey@0"
I20260926 03:32:16.835176 9479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:32:16.851887 9479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:32:16.855786 9479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:32:16.856065 9479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:32:16.857137 9479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:32:16.858376 9479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:32:16.858469 9479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:16.858614 9479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:32:16.858707 9479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:16.891570 9479 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33551
I20260926 03:32:16.891703 9606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33551 every 8 connection(s)
I20260926 03:32:16.893066 9479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:32:16.902275 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 9479
Sep 26 03:32:16 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:16 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 03:32:16.909705 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:16.895577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47123 (local address 127.0.97.62:38695)
0926 03:32:16.895781 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:16.895787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:16.897108 (+ 1321us) server_negotiation.cc:407] Connection header received
0926 03:32:16.898849 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:16.898854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:16.898930 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:16.899021 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:16.900456 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:16.901315 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:16.902840 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:16.903452 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:16.903632 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:16.909444 (+ 5812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:47123: BlockingRecv error: recv got EOF from 127.0.97.3:47123 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:32:16.910547 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:16.896088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.3:47123)
0926 03:32:16.896946 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:16.896989 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:16.898607 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:16.899207 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:16.899214 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:16.899736 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:16.900306 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:16.900323 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:16.901444 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:16.901448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:16.902694 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:16.902705 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:16.903899 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:16.903909 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:16.904176 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:16.904982 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:16.905018 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:16.909254 (+ 4236us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":374,"threads_started":1}
W20260926 03:32:16.911168 9607 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38695 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:38695: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 26 03:32:16 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:16.919586 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:16.903298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34720 (local address 127.0.97.62:38695)
0926 03:32:16.903970 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:16.903975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:16.903990 (+ 15us) server_negotiation.cc:407] Connection header received
0926 03:32:16.904026 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:16.904029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:16.904083 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:16.904168 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:16.906165 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:16.907041 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:16.908716 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:16.909146 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:16.909293 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:16.912473 (+ 3180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:16.912481 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:16.912495 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:16.912526 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:16.916314 (+ 3788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:16.917125 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:16.917135 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:16.917145 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:16.917349 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:16.917751 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:16.917754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:16.917756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:16.918179 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:16.918504 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:16.918858 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 03:32:16.919166 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":469,"threads_started":1}
Sep 26 03:32:16 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:16.920889 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:16.911743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.3:36135)
0926 03:32:16.911889 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:16.911908 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:16.912003 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:16.912284 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:16.912291 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:16.912654 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:16.912785 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:16.912792 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:16.914342 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:16.914346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:16.915353 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:16.915364 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:16.916457 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:16.916463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:16.916680 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:16.917244 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:16.917259 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:16.920641 (+ 3382us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 03:32:16.921077 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:16.911844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36135 (local address 127.0.97.62:38695)
0926 03:32:16.912009 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:16.912013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:16.912028 (+ 15us) server_negotiation.cc:407] Connection header received
0926 03:32:16.912098 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:16.912101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:16.912150 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:16.912225 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:16.912929 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:16.914206 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:16.915500 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:16.916299 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:16.916440 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:16.920819 (+ 4379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:36135: BlockingRecv error: recv got EOF from 127.0.97.3:36135 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 03:32:16 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:16 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:16.929165 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:16.922239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.3:46053)
0926 03:32:16.922418 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:16.922435 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:16.922525 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:16.922825 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:16.922830 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:16.923107 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:16.923235 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:16.923242 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:16.924160 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:16.924166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:16.925147 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:16.925158 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:16.925800 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:16.925806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:16.925962 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:16.926418 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:16.926433 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:16.928970 (+ 2537us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 03:32:16.929381 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:16.922363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46053 (local address 127.0.97.62:38695)
0926 03:32:16.922569 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:16.922574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:16.922589 (+ 15us) server_negotiation.cc:407] Connection header received
0926 03:32:16.922628 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:16.922635 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:16.922699 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:16.922804 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:16.923362 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:16.924029 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:16.925283 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:16.925667 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:16.925792 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:16.929133 (+ 3341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:46053: BlockingRecv error: recv got EOF from 127.0.97.3:46053 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
W20260926 03:32:16.929417 9607 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 03:32:17 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:17 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:17.037748 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:17.029452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.1:58937)
0926 03:32:17.030025 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:17.030044 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:17.030206 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:17.030589 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:17.030597 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:17.031214 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:17.031403 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:17.031411 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:17.032350 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:17.032353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:17.033365 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:17.033375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:17.034059 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:17.034063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:17.034195 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:17.034739 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:17.034755 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:17.037531 (+ 2776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":326,"threads_started":1}
W20260926 03:32:17.037963 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:17.029615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58937 (local address 127.0.97.62:38695)
0926 03:32:17.029784 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:17.029789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:17.030152 (+ 363us) server_negotiation.cc:407] Connection header received
0926 03:32:17.030346 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:17.030354 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:17.030436 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:17.030531 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:17.031527 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:17.032234 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:17.033490 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:17.033887 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:17.034014 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:17.037750 (+ 3736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58937: BlockingRecv error: recv got EOF from 127.0.97.1:58937 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 03:32:17 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:17 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:17.500743 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:17.492684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.2:43831)
0926 03:32:17.493199 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:17.493215 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:17.493325 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:17.493654 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:17.493657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:17.494096 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:17.494286 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:17.494295 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:17.495307 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:17.495312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:17.496275 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:17.496287 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:17.497054 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:17.497058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:17.497185 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:17.497760 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:17.497800 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:17.500434 (+ 2634us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":276,"threads_started":1}
W20260926 03:32:17.501044 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:17.492758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43831 (local address 127.0.97.62:38695)
0926 03:32:17.492954 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:17.492959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:17.493291 (+ 332us) server_negotiation.cc:407] Connection header received
0926 03:32:17.493461 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:17.493465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:17.493529 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:17.493615 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:17.494410 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:17.495155 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:17.496412 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:17.496871 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:17.496962 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:17.500744 (+ 3782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:43831: BlockingRecv error: recv got EOF from 127.0.97.2:43831 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 03:32:17 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:17 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:17.949925 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:17.930225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.3:53475)
0926 03:32:17.930931 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:17.930950 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:17.931094 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:17.931573 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:17.931579 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:17.932217 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:17.932461 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:17.932472 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:17.933581 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:17.933587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:17.935068 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:17.935083 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:17.935964 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:17.935968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:17.936095 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:17.945582 (+ 9487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:17.945621 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:17.949644 (+ 4023us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":370,"threads_started":1}
W20260926 03:32:17.950084 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:17.930271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53475 (local address 127.0.97.62:38695)
0926 03:32:17.930462 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:17.930467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:17.931038 (+ 571us) server_negotiation.cc:407] Connection header received
0926 03:32:17.931242 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:17.931246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:17.931399 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:17.931509 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:17.932663 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:17.933395 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:17.935350 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:17.935773 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:17.935907 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:17.949841 (+ 13934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:53475: BlockingRecv error: recv got EOF from 127.0.97.3:53475 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 26 03:32:18 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:18 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:18.049028 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:18.038981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.1:34863)
0926 03:32:18.039631 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:18.039649 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:18.039809 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:18.040257 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:18.040264 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:18.040825 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:18.041002 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:18.041011 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:18.042168 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:18.042175 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:18.043150 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:18.043162 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:18.043885 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:18.043891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:18.044064 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:18.044742 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:18.044772 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:18.048785 (+ 4013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":356,"threads_started":1}
W20260926 03:32:18.049346 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:18.039136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34863 (local address 127.0.97.62:38695)
0926 03:32:18.039393 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:18.039398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:18.039766 (+ 368us) server_negotiation.cc:407] Connection header received
0926 03:32:18.039953 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:18.039957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:18.040043 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:18.040206 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:18.041170 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:18.042024 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:18.043306 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:18.043723 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:18.043863 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:18.049018 (+ 5155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:34863: BlockingRecv error: recv got EOF from 127.0.97.1:34863 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 26 03:32:18 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:18 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:18.511746 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:18.501706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.2:56339)
0926 03:32:18.502332 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:18.502352 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:18.502500 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:18.502855 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:18.502858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:18.503320 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:18.503515 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:18.503525 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:18.504541 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:18.504547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:18.506083 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:18.506097 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:18.506869 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:18.506875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:18.507048 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:18.507732 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:18.507755 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:18.511386 (+ 3631us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":336,"threads_started":1}
W20260926 03:32:18.511909 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:18.501807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56339 (local address 127.0.97.62:38695)
0926 03:32:18.502008 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:18.502012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:18.502451 (+ 439us) server_negotiation.cc:407] Connection header received
0926 03:32:18.502616 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:18.502620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:18.502708 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:18.502801 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:18.503647 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:18.504388 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:18.506246 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:18.506712 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:18.506818 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:18.511627 (+ 4809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56339: BlockingRecv error: recv got EOF from 127.0.97.2:56339 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 03:32:18 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:18 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:18.959465 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:18.950870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.3:39333)
0926 03:32:18.951450 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:18.951466 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:18.951589 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:18.951985 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:18.951989 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:18.952395 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:18.952625 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:18.952636 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:18.953640 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:18.953644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:18.954572 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:18.954582 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:18.955212 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:18.955216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:18.955323 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:18.955947 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:18.955965 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:18.959178 (+ 3213us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":326,"threads_started":1}
W20260926 03:32:18.959606 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:18.951113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39333 (local address 127.0.97.62:38695)
0926 03:32:18.951290 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:18.951295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:18.951543 (+ 248us) server_negotiation.cc:407] Connection header received
0926 03:32:18.951736 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:18.951740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:18.951810 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:18.951935 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:18.952772 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:18.953518 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:18.954693 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:18.955084 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:18.955175 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:18.959354 (+ 4179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:39333: BlockingRecv error: recv got EOF from 127.0.97.3:39333 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 26 03:32:19 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:19 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:19.058507 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:19.050212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.1:45677)
0926 03:32:19.050736 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:19.050752 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:19.050869 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:19.051164 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:19.051167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:19.051501 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:19.051694 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:19.051702 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:19.052735 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:19.052740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:19.053630 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:19.053641 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:19.054457 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:19.054463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:19.054601 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:19.055388 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:19.055405 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:19.058130 (+ 2725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":279,"threads_started":1}
W20260926 03:32:19.058544 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:19.050248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45677 (local address 127.0.97.62:38695)
0926 03:32:19.050460 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:19.050464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:19.050853 (+ 389us) server_negotiation.cc:407] Connection header received
0926 03:32:19.050972 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:19.050976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:19.051036 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:19.051121 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:19.051828 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:19.052592 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:19.053753 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:19.054287 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:19.054382 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:19.058302 (+ 3920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45677: BlockingRecv error: recv got EOF from 127.0.97.1:45677 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 26 03:32:19 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:19 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:19.521801 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:19.512880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.2:46441)
0926 03:32:19.513415 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:19.513433 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:19.513603 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:19.513994 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:19.513999 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:19.514373 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:19.514560 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:19.514570 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:19.515733 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:19.515739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:19.516812 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:19.516824 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:19.517621 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:19.517627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:19.517816 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:19.518435 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:19.518454 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:19.521394 (+ 2940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":274,"threads_started":1}
W20260926 03:32:19.521874 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:19.512951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46441 (local address 127.0.97.62:38695)
0926 03:32:19.513193 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:19.513202 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:19.513522 (+ 320us) server_negotiation.cc:407] Connection header received
0926 03:32:19.513732 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:19.513739 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:19.513818 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:19.513931 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:19.514710 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:19.515582 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:19.516999 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:19.517468 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:19.517609 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:19.521615 (+ 4006us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46441: BlockingRecv error: recv got EOF from 127.0.97.2:46441 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 03:32:19 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:19 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:19.969475 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:19.960448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.3:55237)
0926 03:32:19.960998 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:19.961014 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:19.961127 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:19.961417 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:19.961420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:19.961739 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:19.961894 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:19.961945 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:19.963092 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:19.963096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:19.964278 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:19.964290 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:19.965339 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:19.965344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:19.965511 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:19.966202 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:19.966219 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:19.969219 (+ 3000us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":293,"threads_started":1}
W20260926 03:32:19.969679 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:19.960612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55237 (local address 127.0.97.62:38695)
0926 03:32:19.960860 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:19.960867 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:19.961094 (+ 227us) server_negotiation.cc:407] Connection header received
0926 03:32:19.961226 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:19.961230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:19.961294 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:19.961378 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:19.962066 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:19.962951 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:19.964427 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:19.965133 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:19.965316 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:19.969453 (+ 4137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:55237: BlockingRecv error: recv got EOF from 127.0.97.3:55237 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 26 03:32:20 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:20 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:20.068508 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:20.059524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.1:58855)
0926 03:32:20.060013 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:20.060029 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:20.060146 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:20.060445 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:20.060448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:20.060836 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:20.061022 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:20.061030 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:20.062038 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:20.062044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:20.063576 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:20.063588 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:20.064347 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:20.064352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:20.064495 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:20.065223 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:20.065240 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:20.068277 (+ 3037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":265,"threads_started":1}
W20260926 03:32:20.068840 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:20.059677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58855 (local address 127.0.97.62:38695)
0926 03:32:20.059938 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:20.059946 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:20.060109 (+ 163us) server_negotiation.cc:407] Connection header received
0926 03:32:20.060256 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:20.060260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:20.060319 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:20.060399 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:20.061153 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:20.061874 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:20.063746 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:20.064209 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:20.064307 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:20.068504 (+ 4197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58855: BlockingRecv error: recv got EOF from 127.0.97.1:58855 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Sep 26 03:32:20 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:20 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:20.531716 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:20.522836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.2:51137)
0926 03:32:20.523380 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:20.523396 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:20.523522 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:20.523844 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:20.523848 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:20.524206 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:20.524344 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:20.524352 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:20.525443 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:20.525447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:20.526426 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:20.526437 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:20.527357 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:20.527362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:20.527507 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:20.528133 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:20.528151 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:20.531381 (+ 3230us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":320,"threads_started":1}
W20260926 03:32:20.531785 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:20.522922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51137 (local address 127.0.97.62:38695)
0926 03:32:20.523139 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:20.523144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:20.523489 (+ 345us) server_negotiation.cc:407] Connection header received
0926 03:32:20.523620 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:20.523624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:20.523683 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:20.523766 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:20.524469 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:20.525315 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:20.526576 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:20.527203 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:20.527298 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:20.531571 (+ 4273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51137: BlockingRecv error: recv got EOF from 127.0.97.2:51137 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 26 03:32:20 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:20 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:20.978652 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:20.970394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.3:33609)
0926 03:32:20.970958 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:20.970974 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:20.971084 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:20.971389 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:20.971392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:20.971744 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:20.971894 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:20.971903 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:20.973020 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:20.973024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:20.973961 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:20.973971 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:20.974592 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:20.974596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:20.974726 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:20.975291 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:20.975307 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:20.978286 (+ 2979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":251,"threads_started":1}
W20260926 03:32:20.978732 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:20.970483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33609 (local address 127.0.97.62:38695)
0926 03:32:20.970709 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:20.970717 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:20.971049 (+ 332us) server_negotiation.cc:407] Connection header received
0926 03:32:20.971172 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:20.971176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:20.971234 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:20.971309 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:20.972036 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:20.972858 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:20.974090 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:20.974473 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:20.974563 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:20.978484 (+ 3921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:33609: BlockingRecv error: recv got EOF from 127.0.97.3:33609 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 26 03:32:21 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:21 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:21.077479 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:21.069505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.1:43631)
0926 03:32:21.070077 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:21.070093 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:21.070211 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:21.070467 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:21.070470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:21.070838 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:21.070991 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:21.070999 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:21.071944 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:21.071947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:21.072859 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:21.072870 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:21.073481 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:21.073485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:21.073596 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:21.074189 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:21.074206 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:21.077130 (+ 2924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":290,"threads_started":1}
W20260926 03:32:21.077767 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:21.069599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43631 (local address 127.0.97.62:38695)
0926 03:32:21.069774 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:21.069778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:21.070172 (+ 394us) server_negotiation.cc:407] Connection header received
0926 03:32:21.070293 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:21.070296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:21.070353 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:21.070429 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:21.071117 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:21.071824 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:21.072989 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:21.073362 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:21.073449 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:21.077515 (+ 4066us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:43631: BlockingRecv error: recv got EOF from 127.0.97.1:43631 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 03:32:21 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:21 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:21.541188 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:21.532840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.2:55711)
0926 03:32:21.533526 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:21.533543 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:21.533650 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:21.533904 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:21.533907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:21.534281 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:21.534429 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:21.534437 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:21.535540 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:21.535544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:21.536449 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:21.536460 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:21.537323 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:21.537326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:21.537443 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:21.538071 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:21.538087 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:21.540851 (+ 2764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":423,"threads_started":1}
W20260926 03:32:21.541306 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:21.533019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55711 (local address 127.0.97.62:38695)
0926 03:32:21.533203 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:21.533208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:21.533619 (+ 411us) server_negotiation.cc:407] Connection header received
0926 03:32:21.533729 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:21.533732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:21.533788 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:21.533866 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:21.534612 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:21.535367 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:21.536644 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:21.537195 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:21.537288 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:21.541061 (+ 3773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:55711: BlockingRecv error: recv got EOF from 127.0.97.2:55711 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 26 03:32:21 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:21 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:21.988663 9626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:21.979650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.3:56721)
0926 03:32:21.980175 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:21.980194 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:21.980311 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:21.980742 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:21.980746 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:21.981197 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:21.981360 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:21.981370 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:21.982429 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:21.982435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:21.983760 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:21.983776 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:21.984628 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:21.984635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:21.984768 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:21.985395 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:21.985413 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:21.988265 (+ 2852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":251,"threads_started":1}
W20260926 03:32:21.988744 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:21.979735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56721 (local address 127.0.97.62:38695)
0926 03:32:21.979947 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:21.979956 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:21.980272 (+ 316us) server_negotiation.cc:407] Connection header received
0926 03:32:21.980472 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:21.980476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:21.980539 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:21.980655 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:21.981521 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:21.982255 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:21.983998 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:21.984437 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:21.984531 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:21.988478 (+ 3947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56721: BlockingRecv error: recv got EOF from 127.0.97.3:56721 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 03:32:22 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:22 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:22.086690 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:22.078488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.1:53805)
0926 03:32:22.079234 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:22.079251 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:22.079401 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:22.079728 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:22.079733 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:22.080128 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:22.080269 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:22.080277 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:22.081365 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:22.081373 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:22.082257 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:22.082267 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:22.082967 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:22.082973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:22.083143 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:22.083721 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:22.083738 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:22.086478 (+ 2740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":467,"threads_started":1}
W20260926 03:32:22.086902 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:22.078571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53805 (local address 127.0.97.62:38695)
0926 03:32:22.078804 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:22.078811 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:22.079330 (+ 519us) server_negotiation.cc:407] Connection header received
0926 03:32:22.079539 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:22.079548 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:22.079605 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:22.079708 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:22.080402 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:22.081232 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:22.082397 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:22.082804 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:22.082927 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:22.086652 (+ 3725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:53805: BlockingRecv error: recv got EOF from 127.0.97.1:53805 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 26 03:32:22 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:22 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:22.550442 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:22.542172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.2:37065)
0926 03:32:22.542698 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:22.542714 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:22.542826 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:22.543110 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:22.543113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:22.543482 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:22.543630 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:22.543638 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:22.544740 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:22.544744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:22.545824 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:22.545837 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:22.546541 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:22.546546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:22.546670 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:22.547247 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:22.547264 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:22.550113 (+ 2849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":303,"threads_started":1}
W20260926 03:32:22.550637 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:22.542219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37065 (local address 127.0.97.62:38695)
0926 03:32:22.542386 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:22.542390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:22.542787 (+ 397us) server_negotiation.cc:407] Connection header received
0926 03:32:22.542918 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:22.542922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:22.542986 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:22.543066 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:22.543762 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:22.544446 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:22.545980 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:22.546375 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:22.546477 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:22.550392 (+ 3915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:37065: BlockingRecv error: recv got EOF from 127.0.97.2:37065 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 03:32:22 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:23 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:23.005915 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:22.989712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.3:47965)
0926 03:32:22.990618 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:22.990634 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:22.990784 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:22.991219 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:22.991223 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:22.991604 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:22.991745 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:22.991752 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:22.993172 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:22.993177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:22.994069 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:22.994080 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:22.994791 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:22.994796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:22.994907 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:23.002280 (+ 7373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:23.002300 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:23.005606 (+ 3306us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":788,"thread_start_us":414,"threads_started":1}
W20260926 03:32:23.006415 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:22.989796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47965 (local address 127.0.97.62:38695)
0926 03:32:22.989988 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:22.989995 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:22.990722 (+ 727us) server_negotiation.cc:407] Connection header received
0926 03:32:22.990988 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:22.990993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:22.991067 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:22.991174 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:22.991887 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:22.993030 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:22.994220 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:22.994663 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:22.994769 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:23.006096 (+ 11327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:47965: BlockingRecv error: recv got EOF from 127.0.97.3:47965 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 26 03:32:23 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:23 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:23.095770 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:23.087560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.1:33875)
0926 03:32:23.088161 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:23.088178 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:23.088284 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:23.088641 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:23.088644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:23.089029 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:23.089187 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:23.089194 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:23.090241 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:23.090246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:23.091245 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:23.091257 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:23.092002 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:23.092007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:23.092120 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:23.092799 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:23.092816 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:23.095518 (+ 2702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":291,"threads_started":1}
W20260926 03:32:23.096051 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:23.087665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33875 (local address 127.0.97.62:38695)
0926 03:32:23.087833 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:23.087838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:23.088250 (+ 412us) server_negotiation.cc:407] Connection header received
0926 03:32:23.088388 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:23.088391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:23.088464 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:23.088539 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:23.089381 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:23.090089 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:23.091399 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:23.091828 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:23.091956 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:23.095762 (+ 3806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:33875: BlockingRecv error: recv got EOF from 127.0.97.1:33875 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 03:32:23 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:23 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:23.560106 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:23.551356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.2:38223)
0926 03:32:23.551888 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:23.551905 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:23.552041 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:23.552321 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:23.552324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:23.552860 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:23.553073 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:23.553081 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:23.554151 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:23.554155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:23.555056 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:23.555066 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:23.555880 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:23.555886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:23.556060 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:23.556852 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:23.556877 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:23.559854 (+ 2977us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":255,"threads_started":1}
W20260926 03:32:23.560289 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:23.551441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38223 (local address 127.0.97.62:38695)
0926 03:32:23.551752 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:23.551758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:23.551988 (+ 230us) server_negotiation.cc:407] Connection header received
0926 03:32:23.552139 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:23.552142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:23.552202 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:23.552280 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:23.553202 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:23.554022 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:23.555226 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:23.555708 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:23.555836 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:23.560057 (+ 4221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:38223: BlockingRecv error: recv got EOF from 127.0.97.2:38223 (error 108)
Metrics: {"server-negotiator.queue_time_us":136}
Sep 26 03:32:24 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:24 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:24.015232 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:24.006915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.3:33153)
0926 03:32:24.007567 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:24.007588 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:24.007723 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:24.008081 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:24.008090 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:24.008689 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:24.008909 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:24.008921 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:24.010038 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:24.010042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:24.010929 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:24.010940 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:24.011599 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:24.011603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:24.011711 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:24.012256 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:24.012272 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:24.014991 (+ 2719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":304,"threads_started":1}
W20260926 03:32:24.015357 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:24.007049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33153 (local address 127.0.97.62:38695)
0926 03:32:24.007274 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:24.007283 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:24.007684 (+ 401us) server_negotiation.cc:407] Connection header received
0926 03:32:24.007833 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:24.007837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:24.007930 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:24.008012 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:24.009070 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:24.009899 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:24.011062 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:24.011477 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:24.011570 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:24.015176 (+ 3606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:33153: BlockingRecv error: recv got EOF from 127.0.97.3:33153 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 03:32:24 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:24 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:24.105173 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:24.096935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.1:57107)
0926 03:32:24.097438 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:24.097465 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:24.097615 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:24.097953 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:24.097958 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:24.098376 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:24.098525 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:24.098533 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:24.099522 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:24.099528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:24.100432 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:24.100443 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:24.101228 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:24.101234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:24.101395 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:24.102049 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:24.102065 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:24.104837 (+ 2772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":256,"threads_started":1}
W20260926 03:32:24.105310 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:24.097175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57107 (local address 127.0.97.62:38695)
0926 03:32:24.097385 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:24.097392 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:24.097550 (+ 158us) server_negotiation.cc:407] Connection header received
0926 03:32:24.097735 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:24.097740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:24.097826 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:24.097925 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:24.098665 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:24.099375 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:24.100602 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:24.101038 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:24.101172 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:24.105053 (+ 3881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:57107: BlockingRecv error: recv got EOF from 127.0.97.1:57107 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 26 03:32:24 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:24 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:24.569947 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:24.561162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.2:54927)
0926 03:32:24.561635 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:24.561694 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:24.561976 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:24.562323 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:24.562328 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:24.562779 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:24.562953 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:24.562962 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:24.564195 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:24.564203 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:24.565216 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:24.565228 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:24.566053 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:24.566060 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:24.566226 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:24.566833 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:24.566849 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:24.569725 (+ 2876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":246,"threads_started":1}
W20260926 03:32:24.570205 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:24.561243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54927 (local address 127.0.97.62:38695)
0926 03:32:24.561469 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:24.561476 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:24.561779 (+ 303us) server_negotiation.cc:407] Connection header received
0926 03:32:24.562125 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:24.562130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:24.562198 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:24.562299 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:24.563094 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:24.564033 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:24.565367 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:24.565861 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:24.566015 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:24.569981 (+ 3966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:54927: BlockingRecv error: recv got EOF from 127.0.97.2:54927 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 26 03:32:25 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:25 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:25.025816 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:25.016307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.3:55853)
0926 03:32:25.016834 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:25.016852 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:25.017014 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:25.017317 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:25.017321 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:25.017731 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:25.017920 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:25.017931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:25.018945 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:25.018949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:25.019882 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:25.019894 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:25.020930 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:25.020937 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:25.021120 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:25.021894 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:25.021920 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:25.025470 (+ 3550us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":319,"threads_started":1}
W20260926 03:32:25.025976 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:25.016475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55853 (local address 127.0.97.62:38695)
0926 03:32:25.016740 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:25.016747 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:25.016977 (+ 230us) server_negotiation.cc:407] Connection header received
0926 03:32:25.017118 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:25.017122 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:25.017185 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:25.017267 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:25.018053 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:25.018772 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:25.020035 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:25.020676 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:25.020832 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:25.025688 (+ 4856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:55853: BlockingRecv error: recv got EOF from 127.0.97.3:55853 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Sep 26 03:32:25 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:25 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:25.115942 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:25.106382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.1:59743)
0926 03:32:25.107153 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:25.107199 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:25.107393 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:25.107781 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:25.107786 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:25.108267 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:25.108460 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:25.108469 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:25.109580 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:25.109587 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:25.110558 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:25.110569 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:25.111345 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:25.111350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:25.111535 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:25.112251 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:25.112269 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:25.115550 (+ 3281us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":460,"threads_started":1}
W20260926 03:32:25.116024 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:25.106570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59743 (local address 127.0.97.62:38695)
0926 03:32:25.106877 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:25.106885 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:25.107305 (+ 420us) server_negotiation.cc:407] Connection header received
0926 03:32:25.107542 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:25.107549 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:25.107638 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:25.107748 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:25.108649 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:25.109437 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:25.110701 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:25.111186 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:25.111317 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:25.115767 (+ 4450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:59743: BlockingRecv error: recv got EOF from 127.0.97.1:59743 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Sep 26 03:32:25 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:25 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:25.580354 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:25.571217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.2:45341)
0926 03:32:25.571846 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:25.571865 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:25.571993 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:25.572362 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:25.572365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:25.572771 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:25.572962 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:25.572971 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:25.574122 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:25.574126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:25.575400 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:25.575416 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:25.576274 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:25.576280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:25.576454 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:25.577207 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:25.577226 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:25.580000 (+ 2774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":343,"threads_started":1}
W20260926 03:32:25.580490 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:25.571249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45341 (local address 127.0.97.62:38695)
0926 03:32:25.571536 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:25.571550 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:25.571990 (+ 440us) server_negotiation.cc:407] Connection header received
0926 03:32:25.572112 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:25.572117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:25.572213 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:25.572320 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:25.573138 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:25.573988 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:25.575593 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:25.576109 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:25.576259 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:25.580227 (+ 3968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:45341: BlockingRecv error: recv got EOF from 127.0.97.2:45341 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 26 03:32:26 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:26 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:26.037039 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:26.027109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.3:42887)
0926 03:32:26.027733 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:26.027751 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:26.027893 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:26.028341 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:26.028345 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:26.028995 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:26.029220 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:26.029230 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:26.030517 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:26.030522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:26.031774 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:26.031787 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:26.032512 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:26.032517 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:26.032742 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:26.033374 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:26.033393 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:26.036727 (+ 3334us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":351,"threads_started":1}
W20260926 03:32:26.037153 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:26.027181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42887 (local address 127.0.97.62:38695)
0926 03:32:26.027390 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:26.027395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:26.027862 (+ 467us) server_negotiation.cc:407] Connection header received
0926 03:32:26.028038 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:26.028043 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:26.028114 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:26.028224 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:26.029377 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:26.030358 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:26.031910 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:26.032334 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:26.032447 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:26.036952 (+ 4505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:42887: BlockingRecv error: recv got EOF from 127.0.97.3:42887 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 26 03:32:26 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:26 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:26.125201 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:26.117074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.1:41399)
0926 03:32:26.117588 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:26.117605 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:26.117712 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:26.117994 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:26.117997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:26.118404 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:26.118564 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:26.118573 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:26.119588 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:26.119592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:26.120526 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:26.120536 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:26.121279 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:26.121283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:26.121411 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:26.122035 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:26.122053 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:26.124953 (+ 2900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":297,"threads_started":1}
W20260926 03:32:26.125399 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:26.117129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41399 (local address 127.0.97.62:38695)
0926 03:32:26.117296 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:26.117301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:26.117676 (+ 375us) server_negotiation.cc:407] Connection header received
0926 03:32:26.117804 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:26.117808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:26.117878 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:26.117958 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:26.118718 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:26.119453 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:26.120703 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:26.121125 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:26.121246 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:26.125150 (+ 3904us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:41399: BlockingRecv error: recv got EOF from 127.0.97.1:41399 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 03:32:26 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:26 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:26.589772 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:26.581474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.2:43149)
0926 03:32:26.581973 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:26.581988 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:26.582157 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:26.582530 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:26.582559 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:26.582925 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:26.583060 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:26.583068 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:26.584017 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:26.584024 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:26.585008 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:26.585019 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:26.585679 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:26.585685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:26.585897 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:26.586592 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:26.586610 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:26.589563 (+ 2953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":282,"threads_started":1}
W20260926 03:32:26.590061 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:26.581556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43149 (local address 127.0.97.62:38695)
0926 03:32:26.581784 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:26.581792 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:26.582091 (+ 299us) server_negotiation.cc:407] Connection header received
0926 03:32:26.582285 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:26.582296 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:26.582392 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:26.582501 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:26.583194 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:26.583893 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:26.585144 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:26.585531 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:26.585662 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:26.589780 (+ 4118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:43149: BlockingRecv error: recv got EOF from 127.0.97.2:43149 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 03:32:27 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:27 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:27.046214 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:27.038077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.3:41045)
0926 03:32:27.038585 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:27.038607 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:27.038718 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:27.039080 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:27.039085 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:27.039503 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:27.039706 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:27.039717 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:27.040757 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:27.040761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:27.041696 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:27.041707 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:27.042363 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:27.042367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:27.042485 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:27.043083 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:27.043101 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:27.045914 (+ 2813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":260,"threads_started":1}
W20260926 03:32:27.046432 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:27.038250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41045 (local address 127.0.97.62:38695)
0926 03:32:27.038474 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:27.038480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:27.038680 (+ 200us) server_negotiation.cc:407] Connection header received
0926 03:32:27.038850 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:27.038856 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:27.038919 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:27.038996 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:27.039829 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:27.040615 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:27.041815 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:27.042225 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:27.042318 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:27.046178 (+ 3860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:41045: BlockingRecv error: recv got EOF from 127.0.97.3:41045 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 26 03:32:27 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:27 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:27.134529 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:27.126177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.1:56353)
0926 03:32:27.126614 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:27.126630 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:27.126739 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:27.127029 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:27.127032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:27.127408 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:27.127580 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:27.127610 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:27.128678 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:27.128684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:27.129925 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:27.129937 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:27.130650 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:27.130655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:27.130814 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:27.131512 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:27.131528 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:27.134187 (+ 2659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":227,"threads_started":1}
W20260926 03:32:27.134588 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:27.126236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56353 (local address 127.0.97.62:38695)
0926 03:32:27.126467 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:27.126473 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:27.126729 (+ 256us) server_negotiation.cc:407] Connection header received
0926 03:32:27.126837 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:27.126840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:27.126910 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:27.126987 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:27.127746 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:27.128494 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:27.130052 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:27.130495 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:27.130631 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:27.134367 (+ 3736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:56353: BlockingRecv error: recv got EOF from 127.0.97.1:56353 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 03:32:27 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:27 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:27.598727 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:27.590638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.2:48475)
0926 03:32:27.591156 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:27.591173 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:27.591283 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:27.591589 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:27.591592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:27.592029 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:27.592198 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:27.592207 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:27.593357 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:27.593365 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:27.594474 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:27.594486 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:27.595108 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:27.595112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:27.595228 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:27.595816 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:27.595833 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:27.598498 (+ 2665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":300,"threads_started":1}
W20260926 03:32:27.598966 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:27.590813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48475 (local address 127.0.97.62:38695)
0926 03:32:27.590984 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:27.590988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:27.591250 (+ 262us) server_negotiation.cc:407] Connection header received
0926 03:32:27.591382 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:27.591385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:27.591442 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:27.591523 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:27.592330 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:27.593218 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:27.594596 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:27.594973 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:27.595086 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:27.598707 (+ 3621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:48475: BlockingRecv error: recv got EOF from 127.0.97.2:48475 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 03:32:28 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:28 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:28.060017 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:28.047192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.3:58339)
0926 03:32:28.047645 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:28.047661 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:28.047770 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:28.048044 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:28.048047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:28.048439 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:28.048625 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:28.048633 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:28.049571 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:28.049575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:28.050475 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:28.050486 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:28.051114 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:28.051119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:28.051238 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:28.057025 (+ 5787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:28.057044 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:28.059792 (+ 2748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":265,"threads_started":1}
W20260926 03:32:28.060171 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:28.047246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58339 (local address 127.0.97.62:38695)
0926 03:32:28.047481 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:28.047487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:28.047738 (+ 251us) server_negotiation.cc:407] Connection header received
0926 03:32:28.047863 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:28.047867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:28.047923 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:28.047997 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:28.048742 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:28.049460 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:28.050598 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:28.050960 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:28.051059 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:28.059969 (+ 8910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58339: BlockingRecv error: recv got EOF from 127.0.97.3:58339 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 26 03:32:28 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:28 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:28.143968 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:28.135527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.1:33501)
0926 03:32:28.136903 (+ 1376us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:28.136919 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:28.137035 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:28.137312 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:28.137315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:28.137666 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:28.137813 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:28.137820 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:28.138858 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:28.138862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:28.139840 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:28.139850 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:28.140523 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:28.140527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:28.140661 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:28.141226 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:28.141242 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:28.143750 (+ 2508us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1284,"thread_start_us":1190,"threads_started":1}
W20260926 03:32:28.144164 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:28.135595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33501 (local address 127.0.97.62:38695)
0926 03:32:28.135797 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:28.135807 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:28.137001 (+ 1194us) server_negotiation.cc:407] Connection header received
0926 03:32:28.137122 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:28.137125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:28.137195 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:28.137273 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:28.137930 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:28.138740 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:28.139953 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:28.140406 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:28.140500 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:28.143922 (+ 3422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:33501: BlockingRecv error: recv got EOF from 127.0.97.1:33501 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 03:32:28 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:28 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:28.607991 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:28.599577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.2:35447)
0926 03:32:28.600087 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:28.600104 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:28.600206 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:28.600469 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:28.600472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:28.600927 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:28.601076 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:28.601084 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:28.601998 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:28.602003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:28.603415 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:28.603447 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:28.604140 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:28.604144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:28.604264 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:28.604855 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:28.604887 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:28.607669 (+ 2782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":301,"threads_started":1}
W20260926 03:32:28.608078 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:28.599649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35447 (local address 127.0.97.62:38695)
0926 03:32:28.599852 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:28.599857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:28.600176 (+ 319us) server_negotiation.cc:407] Connection header received
0926 03:32:28.600298 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:28.600301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:28.600354 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:28.600431 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:28.601201 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:28.601887 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:28.603611 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:28.604011 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:28.604110 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:28.607869 (+ 3759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:35447: BlockingRecv error: recv got EOF from 127.0.97.2:35447 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 03:32:29 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:29 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:29.069123 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:29.061114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.3:60489)
0926 03:32:29.061574 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:29.061589 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:29.061739 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:29.062045 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:29.062058 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:29.062478 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:29.062624 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:29.062633 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:29.063727 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:29.063734 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:29.064656 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:29.064667 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:29.065387 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:29.065393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:29.065569 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:29.066175 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:29.066191 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:29.068768 (+ 2577us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":268,"threads_started":1}
W20260926 03:32:29.069221 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:29.061135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60489 (local address 127.0.97.62:38695)
0926 03:32:29.061329 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:29.061337 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:29.061672 (+ 335us) server_negotiation.cc:407] Connection header received
0926 03:32:29.061853 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:29.061865 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:29.061922 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:29.062022 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:29.062773 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:29.063589 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:29.064794 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:29.065234 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:29.065371 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:29.068968 (+ 3597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:60489: BlockingRecv error: recv got EOF from 127.0.97.3:60489 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 03:32:29 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:29 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:29.153262 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:29.144992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.1:47885)
0926 03:32:29.145516 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:29.145532 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:29.145686 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:29.146023 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:29.146027 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:29.146432 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:29.146605 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:29.146613 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:29.147618 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:29.147626 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:29.148580 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:29.148592 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:29.149314 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:29.149320 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:29.149483 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:29.150131 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:29.150148 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:29.152899 (+ 2751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":258,"threads_started":1}
W20260926 03:32:29.153333 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:29.145104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47885 (local address 127.0.97.62:38695)
0926 03:32:29.145320 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:29.145326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:29.145619 (+ 293us) server_negotiation.cc:407] Connection header received
0926 03:32:29.145806 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:29.145812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:29.145895 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:29.145998 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:29.146749 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:29.147486 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:29.148727 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:29.149136 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:29.149275 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:29.153080 (+ 3805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:47885: BlockingRecv error: recv got EOF from 127.0.97.1:47885 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 03:32:29 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:29 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:29.617655 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:29.609121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.2:44163)
0926 03:32:29.609662 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:29.609677 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:29.609819 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:29.610165 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:29.610170 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:29.610591 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:29.610754 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:29.610762 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:29.611777 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:29.611783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:29.613253 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:29.613265 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:29.613953 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:29.613957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:29.614073 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:29.614632 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:29.614650 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:29.617412 (+ 2762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":289,"threads_started":1}
W20260926 03:32:29.617872 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:29.609286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44163 (local address 127.0.97.62:38695)
0926 03:32:29.609449 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:29.609456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:29.609757 (+ 301us) server_negotiation.cc:407] Connection header received
0926 03:32:29.609948 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:29.609953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:29.610036 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:29.610135 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:29.610887 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:29.611602 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:29.613382 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:29.613791 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:29.613883 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:29.617623 (+ 3740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:44163: BlockingRecv error: recv got EOF from 127.0.97.2:44163 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 03:32:30 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:30 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:30.078822 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:30.070239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.3:40381)
0926 03:32:30.070764 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:30.070779 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:30.070968 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:30.071296 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:30.071300 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:30.071721 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:30.071886 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:30.071896 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:30.072980 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:30.072986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:30.073885 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:30.073896 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:30.074737 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:30.074743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:30.074919 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:30.075597 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:30.075629 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:30.078592 (+ 2963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":272,"threads_started":1}
W20260926 03:32:30.078984 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:30.070342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40381 (local address 127.0.97.62:38695)
0926 03:32:30.070609 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:30.070617 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:30.070873 (+ 256us) server_negotiation.cc:407] Connection header received
0926 03:32:30.071100 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:30.071105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:30.071170 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:30.071271 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:30.072033 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:30.072846 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:30.074038 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:30.074566 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:30.074701 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:30.078776 (+ 4075us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:40381: BlockingRecv error: recv got EOF from 127.0.97.3:40381 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
Sep 26 03:32:30 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:30 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:30.162847 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:30.154267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.1:36307)
0926 03:32:30.154738 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:30.154754 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:30.154861 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:30.155173 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:30.155176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:30.155529 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:30.155683 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:30.155690 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:30.156973 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:30.157055 (+ 82us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:30.157958 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:30.157969 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:30.158738 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:30.158746 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:30.158882 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:30.159618 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:30.159645 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:30.162489 (+ 2844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":264,"threads_started":1}
W20260926 03:32:30.162909 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:30.154321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36307 (local address 127.0.97.62:38695)
0926 03:32:30.154608 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:30.154614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:30.154827 (+ 213us) server_negotiation.cc:407] Connection header received
0926 03:32:30.154953 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:30.154957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:30.155014 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:30.155095 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:30.155805 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:30.156841 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:30.158086 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:30.158594 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:30.158693 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:30.162666 (+ 3973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:36307: BlockingRecv error: recv got EOF from 127.0.97.1:36307 (error 108)
Metrics: {"server-negotiator.queue_time_us":175}
Sep 26 03:32:30 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:30 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:30.626086 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:30.618557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.2:56425)
0926 03:32:30.619039 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:30.619056 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:30.619163 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:30.619433 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:30.619436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:30.619796 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:30.619937 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:30.619944 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:30.620879 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:30.620883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:30.621773 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:30.621783 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:30.622417 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:30.622420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:30.622539 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:30.623191 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:30.623209 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:30.625840 (+ 2631us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":277,"threads_started":1}
W20260926 03:32:30.626171 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:30.618671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56425 (local address 127.0.97.62:38695)
0926 03:32:30.618837 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:30.618841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:30.619129 (+ 288us) server_negotiation.cc:407] Connection header received
0926 03:32:30.619255 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:30.619259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:30.619317 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:30.619396 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:30.620053 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:30.620758 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:30.621887 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:30.622306 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:30.622394 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:30.626008 (+ 3614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56425: BlockingRecv error: recv got EOF from 127.0.97.2:56425 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 03:32:31 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:31 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:31.088630 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:31.079894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.3:35477)
0926 03:32:31.080339 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:31.080362 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:31.080503 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:31.080896 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:31.080900 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:31.081266 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:31.081429 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:31.081437 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:31.082442 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:31.082447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:31.083381 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:31.083401 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:31.084036 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:31.084039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:31.084162 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:31.084808 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:31.084826 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:31.088200 (+ 3374us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":244,"threads_started":1}
W20260926 03:32:31.088708 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:31.079979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35477 (local address 127.0.97.62:38695)
0926 03:32:31.080215 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:31.080222 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:31.080463 (+ 241us) server_negotiation.cc:407] Connection header received
0926 03:32:31.080700 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:31.080704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:31.080766 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:31.080849 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:31.081583 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:31.082289 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:31.083510 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:31.083915 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:31.084005 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:31.088432 (+ 4427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:35477: BlockingRecv error: recv got EOF from 127.0.97.3:35477 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 03:32:31 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:31 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:31.172722 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:31.163996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.1:36805)
0926 03:32:31.164456 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:31.164473 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:31.164643 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:31.164929 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:31.164932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:31.165329 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:31.165494 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:31.165502 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:31.166661 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:31.166665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:31.167669 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:31.167680 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:31.168469 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:31.168473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:31.168705 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:31.169330 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:31.169348 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:31.172330 (+ 2982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":272,"threads_started":1}
W20260926 03:32:31.172878 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:31.164065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36805 (local address 127.0.97.62:38695)
0926 03:32:31.164284 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:31.164293 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:31.164574 (+ 281us) server_negotiation.cc:407] Connection header received
0926 03:32:31.164745 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:31.164749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:31.164806 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:31.164896 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:31.165673 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:31.166502 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:31.167818 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:31.168293 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:31.168406 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:31.172595 (+ 4189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:36805: BlockingRecv error: recv got EOF from 127.0.97.1:36805 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 03:32:31 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:32:31 dist-test-slave-dt7p krb5kdc[8610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:32:31.635799 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:31.627171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38695 (local address 127.0.97.2:39651)
0926 03:32:31.627653 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:31.627670 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:31.627789 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:31.628094 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:31.628097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:31.628487 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:31.628719 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:31.628729 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:31.629865 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:31.629869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:31.630945 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:31.630958 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:31.632038 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:31.632044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:31.632188 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:31.632848 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:31.632866 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:31.635552 (+ 2686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:38695: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":288,"threads_started":1}
W20260926 03:32:31.635952 9346 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:31.627333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39651 (local address 127.0.97.62:38695)
0926 03:32:31.627576 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:31.627582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:31.627757 (+ 175us) server_negotiation.cc:407] Connection header received
0926 03:32:31.627870 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:31.627874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:31.627967 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:31.628049 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:31.628868 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:31.629723 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:31.631086 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:31.631761 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:31.631932 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:31.635725 (+ 3793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:39651: BlockingRecv error: recv got EOF from 127.0.97.2:39651 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 03:32:31.909631 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 9197
I20260926 03:32:31.921309 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:31.910572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50666 (local address 127.0.97.1:40327)
0926 03:32:31.911088 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:31.911095 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:31.911122 (+ 27us) server_negotiation.cc:407] Connection header received
0926 03:32:31.911179 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:31.911186 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:31.911403 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:31.911556 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:31.912971 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:31.914081 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:31.915252 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:31.915799 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:31.915895 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:31.918249 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:31.918258 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:31.918272 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:31.918305 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:31.920820 (+ 2515us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 03:32:31.921088 (+ 268us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50666: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":233,"threads_started":1}
W20260926 03:32:31.921433 9657 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50666: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 03:32:31.921483 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 9197: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:40327: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 03:32:31.921695 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 9197
I20260926 03:32:31.938724 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 9348
I20260926 03:32:31.950299 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:31.939809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37280 (local address 127.0.97.2:42163)
0926 03:32:31.940422 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:31.940429 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:31.940459 (+ 30us) server_negotiation.cc:407] Connection header received
0926 03:32:31.940516 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:31.940523 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:31.940811 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:31.940963 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:31.942133 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:31.943170 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:31.944265 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:31.944848 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:31.944957 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:31.947254 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:31.947262 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:31.947275 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:31.947308 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:31.949786 (+ 2478us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 03:32:31.950022 (+ 236us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37280: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":326,"threads_started":1}
W20260926 03:32:31.950402 9658 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37280: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 03:32:31.950479 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 9348: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:42163: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 03:32:31.950657 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 9348
I20260926 03:32:31.967518 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 9479
I20260926 03:32:31.979247 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:31.968422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58936 (local address 127.0.97.3:33551)
0926 03:32:31.969073 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:31.969080 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:31.969108 (+ 28us) server_negotiation.cc:407] Connection header received
0926 03:32:31.969177 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:31.969183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:31.969394 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:31.969549 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:31.970845 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:31.971910 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:31.973249 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:31.973792 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:31.973890 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:31.976306 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:31.976314 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:31.976327 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:31.976362 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:31.978785 (+ 2423us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 03:32:31.979017 (+ 232us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58936: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":363,"threads_started":1}
W20260926 03:32:31.979360 9659 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58936: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 03:32:31.979386 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 9479: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:33551: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 03:32:31.979496 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 9479
I20260926 03:32:31.996500 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 9128
I20260926 03:32:31.997835 9190 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:32:32.111900 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 9128
2026-09-26T03:32:32Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20992 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T03:32:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:32:32Z Disabled control of system clock
I20260926 03:32:32.185458 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40133
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:41559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40133
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 03:32:32.515987 9664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:32:32.516248 9664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:32:32.516378 9664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:32:32.523543 9664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:32:32.523648 9664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:32:32.523720 9664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:32:32.523809 9664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:32:32.523886 9664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:32:32.533118 9664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40133
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:32:32.534428 9664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:32:32.536126 9664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:32:32.547668 9672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:32.547783 9669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:32.547755 9670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:32:32.548623 9664 server_base.cc:1034] running on GCE node
I20260926 03:32:32.549316 9664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:32:32.550555 9664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:32:32.551753 9664 hybrid_clock.cc:648] HybridClock initialized: now 1790393552551709 us; error 36 us; skew 500 ppm
I20260926 03:32:32.552039 9664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:32:32.554148 9664 webserver.cc:460] Webserver started at http://127.0.97.62:33085/ using document root <none> and password file <none>
I20260926 03:32:32.554935 9664 fs_manager.cc:359] Metadata directory not provided
I20260926 03:32:32.555066 9664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:32:32.555482 9664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:32:32.558359 9664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "0411535716b04401ac5012bf04b5c9fa"
format_stamp: "Formatted at 2026-09-26 03:32:32 on dist-test-slave-dt7p"
server_key: "cf60cc7b70ea5b218a4cfc4b235f72c1"
server_key_iv: "8dd626d32b82efe52ea5402c91842ca4"
server_key_version: "encryptionkey@0"
I20260926 03:32:32.559098 9664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "0411535716b04401ac5012bf04b5c9fa"
format_stamp: "Formatted at 2026-09-26 03:32:32 on dist-test-slave-dt7p"
server_key: "cf60cc7b70ea5b218a4cfc4b235f72c1"
server_key_iv: "8dd626d32b82efe52ea5402c91842ca4"
server_key_version: "encryptionkey@0"
I20260926 03:32:32.563802 9664 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260926 03:32:32.567304 9678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:32.568715 9664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:32:32.568895 9664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "0411535716b04401ac5012bf04b5c9fa"
format_stamp: "Formatted at 2026-09-26 03:32:32 on dist-test-slave-dt7p"
server_key: "cf60cc7b70ea5b218a4cfc4b235f72c1"
server_key_iv: "8dd626d32b82efe52ea5402c91842ca4"
server_key_version: "encryptionkey@0"
I20260926 03:32:32.569110 9664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:32:32.591849 9664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:32:32.592756 9664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:32:32.593055 9664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:32:32.610841 9664 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40133
I20260926 03:32:32.610863 9729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40133 every 8 connection(s)
I20260926 03:32:32.612146 9664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:32:32.615705 9730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:32:32.615964 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 9664
I20260926 03:32:32.616168 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:32:32.616855 388 external_mini_cluster.cc:1428] Setting key e54ae6515ac0710ba066d661097558eb
I20260926 03:32:32.627156 9730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0411535716b04401ac5012bf04b5c9fa: Bootstrap starting.
I20260926 03:32:32.629576 9733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:32.618782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56088 (local address 127.0.97.62:40133)
0926 03:32:32.619660 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:32.619670 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:32.619708 (+ 38us) server_negotiation.cc:407] Connection header received
0926 03:32:32.621602 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:32.621622 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:32.622042 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:32.622396 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:32.622908 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:32.624128 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:32.625526 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:32.626155 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:32.626338 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:32.627533 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:32.627540 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:32.627552 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:32:32.628031 (+ 479us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:32.628099 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:32:32.628119 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:32.628238 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:32.628467 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 03:32:32.628760 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":290,"threads_started":1}
I20260926 03:32:32.630306 9730 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0411535716b04401ac5012bf04b5c9fa: Neither blocks nor log segments found. Creating new log.
I20260926 03:32:32.631398 9730 log.cc:824] T 00000000000000000000000000000000 P 0411535716b04401ac5012bf04b5c9fa: Log is configured to *not* fsync() on all Append() calls
I20260926 03:32:32.634014 9730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0411535716b04401ac5012bf04b5c9fa: No bootstrap required, opened a new log
I20260926 03:32:32.640947 9730 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0411535716b04401ac5012bf04b5c9fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0411535716b04401ac5012bf04b5c9fa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40133 } }
I20260926 03:32:32.641304 9730 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0411535716b04401ac5012bf04b5c9fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:32:32.641400 9730 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0411535716b04401ac5012bf04b5c9fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0411535716b04401ac5012bf04b5c9fa, State: Initialized, Role: FOLLOWER
I20260926 03:32:32.641995 9730 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0411535716b04401ac5012bf04b5c9fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0411535716b04401ac5012bf04b5c9fa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40133 } }
I20260926 03:32:32.642215 9730 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0411535716b04401ac5012bf04b5c9fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:32:32.642364 9730 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0411535716b04401ac5012bf04b5c9fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:32:32.642536 9730 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0411535716b04401ac5012bf04b5c9fa [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:32:32.644625 9730 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0411535716b04401ac5012bf04b5c9fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0411535716b04401ac5012bf04b5c9fa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40133 } }
I20260926 03:32:32.645143 9730 leader_election.cc:302] T 00000000000000000000000000000000 P 0411535716b04401ac5012bf04b5c9fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0411535716b04401ac5012bf04b5c9fa; no voters:
I20260926 03:32:32.645682 9730 leader_election.cc:288] T 00000000000000000000000000000000 P 0411535716b04401ac5012bf04b5c9fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:32:32.645874 9735 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0411535716b04401ac5012bf04b5c9fa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:32:32.646931 9735 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0411535716b04401ac5012bf04b5c9fa [term 1 LEADER]: Becoming Leader. State: Replica: 0411535716b04401ac5012bf04b5c9fa, State: Running, Role: LEADER
I20260926 03:32:32.647475 9735 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0411535716b04401ac5012bf04b5c9fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0411535716b04401ac5012bf04b5c9fa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40133 } }
I20260926 03:32:32.647719 9730 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0411535716b04401ac5012bf04b5c9fa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:32:32.650259 9737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0411535716b04401ac5012bf04b5c9fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0411535716b04401ac5012bf04b5c9fa. Latest consensus state: current_term: 1 leader_uuid: "0411535716b04401ac5012bf04b5c9fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0411535716b04401ac5012bf04b5c9fa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40133 } } }
I20260926 03:32:32.650632 9737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0411535716b04401ac5012bf04b5c9fa [sys.catalog]: This master's current role is: LEADER
I20260926 03:32:32.650389 9736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0411535716b04401ac5012bf04b5c9fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0411535716b04401ac5012bf04b5c9fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0411535716b04401ac5012bf04b5c9fa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40133 } } }
I20260926 03:32:32.650797 9736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0411535716b04401ac5012bf04b5c9fa [sys.catalog]: This master's current role is: LEADER
I20260926 03:32:32.653024 9742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:32:32.658708 9742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:32:32.667513 9742 catalog_manager.cc:1347] Generated new cluster ID: 16794132fc1d4331ae97b4fa4957232b
I20260926 03:32:32.667639 9742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:32:32.681687 9742 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:32:32.682997 9742 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:32:32.689301 9742 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0411535716b04401ac5012bf04b5c9fa: Generated new TSK 0
I20260926 03:32:32.690109 9742 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:32:32.701936 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40133
--builtin_ntp_servers=127.0.97.20:41559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 03:32:33.041499 9754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:32:33.041776 9754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:32:33.041860 9754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:32:33.049268 9754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:32:33.049363 9754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:32:33.049474 9754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:32:33.058677 9754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:32:33.060218 9754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:32:33.062266 9754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:32:33.075518 9759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:33.075589 9762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:33.075563 9760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:32:33.078121 9754 server_base.cc:1034] running on GCE node
I20260926 03:32:33.078889 9754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:32:33.084833 9754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:32:33.086097 9754 hybrid_clock.cc:648] HybridClock initialized: now 1790393553086041 us; error 50 us; skew 500 ppm
I20260926 03:32:33.086449 9754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:32:33.096686 9754 webserver.cc:460] Webserver started at http://127.0.97.1:38943/ using document root <none> and password file <none>
I20260926 03:32:33.097344 9754 fs_manager.cc:359] Metadata directory not provided
I20260926 03:32:33.097429 9754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:32:33.097698 9754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:32:33.100385 9754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-0/data/instance:
uuid: "502b8bcdeefd4ec7bf55afc8deed443d"
format_stamp: "Formatted at 2026-09-26 03:32:33 on dist-test-slave-dt7p"
server_key: "1144133ac55743868090f780de3fad79"
server_key_iv: "686d44effab3467393feb38a057815e7"
server_key_version: "encryptionkey@0"
I20260926 03:32:33.101166 9754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-0/wal/instance:
uuid: "502b8bcdeefd4ec7bf55afc8deed443d"
format_stamp: "Formatted at 2026-09-26 03:32:33 on dist-test-slave-dt7p"
server_key: "1144133ac55743868090f780de3fad79"
server_key_iv: "686d44effab3467393feb38a057815e7"
server_key_version: "encryptionkey@0"
I20260926 03:32:33.106055 9754 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 03:32:33.109306 9768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:33.110563 9754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:32:33.110714 9754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-0/wal
uuid: "502b8bcdeefd4ec7bf55afc8deed443d"
format_stamp: "Formatted at 2026-09-26 03:32:33 on dist-test-slave-dt7p"
server_key: "1144133ac55743868090f780de3fad79"
server_key_iv: "686d44effab3467393feb38a057815e7"
server_key_version: "encryptionkey@0"
I20260926 03:32:33.110934 9754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:32:33.123411 9754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:32:33.124200 9754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:32:33.124466 9754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:32:33.125409 9754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:32:33.126765 9754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:32:33.126852 9754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:33.126996 9754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:32:33.127077 9754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:33.159816 9754 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45455
I20260926 03:32:33.159863 9880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45455 every 8 connection(s)
I20260926 03:32:33.161146 9754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:32:33.168411 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 9754
I20260926 03:32:33.168634 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:32:33.169426 388 external_mini_cluster.cc:1428] Setting key 3b6e3910ef7d69acaabaddaaf4158753
I20260926 03:32:33.172230 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40133
--builtin_ntp_servers=127.0.97.20:41559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 03:32:33.173772 9883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:33.163553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34071 (local address 127.0.97.62:40133)
0926 03:32:33.163982 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:33.163988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:33.165207 (+ 1219us) server_negotiation.cc:407] Connection header received
0926 03:32:33.166826 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:33.166830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:33.166894 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:33.166987 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:33.167935 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:33.168783 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:33.170855 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:33.171376 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:33.171505 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:33.172930 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:33.172935 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:33.172938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:32:33.172954 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:33.172989 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:32:33.173000 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:33.173096 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:33.173315 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 03:32:33.173402 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":199,"threads_started":1}
I20260926 03:32:33.174690 9884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:33.164148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40133 (local address 127.0.97.1:34071)
0926 03:32:33.165054 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:33.165090 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:33.166591 (+ 1501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:33.167185 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:33.167193 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:33.167274 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:33.167797 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:33.167811 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:33.168930 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:33.168937 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:33.170716 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:33.170729 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:33.171584 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:33.171594 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:33.171821 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:33.172613 (+ 792us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:32:33.172617 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:33.172676 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:32:33.172745 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:32:33.173106 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:32:33.173118 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:32:33.173126 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 03:32:33.173312 (+ 186us) client_negotiation.cc:240] Negotiation successful
0926 03:32:33.173693 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":262,"threads_started":1}
I20260926 03:32:33.176075 9881 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40133
I20260926 03:32:33.176388 9881 heartbeater.cc:464] Registering TS with master...
I20260926 03:32:33.177282 9881 heartbeater.cc:511] Master 127.0.97.62:40133 requested a full tablet report, sending...
I20260926 03:32:33.180349 9695 ts_manager.cc:194] Registered new tserver with Master: 502b8bcdeefd4ec7bf55afc8deed443d (127.0.97.1:45455)
I20260926 03:32:33.181932 9695 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:34071
W20260926 03:32:33.497644 9885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:32:33.497946 9885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:32:33.498107 9885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:32:33.505375 9885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:32:33.505527 9885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:32:33.505717 9885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:32:33.514917 9885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:32:33.516179 9885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:32:33.518042 9885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:32:33.530992 9893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:33.531069 9890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:33.531046 9891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:32:33.531795 9885 server_base.cc:1034] running on GCE node
I20260926 03:32:33.532405 9885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:32:33.533371 9885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:32:33.534581 9885 hybrid_clock.cc:648] HybridClock initialized: now 1790393553534522 us; error 58 us; skew 500 ppm
I20260926 03:32:33.534896 9885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:32:33.537233 9885 webserver.cc:460] Webserver started at http://127.0.97.2:35237/ using document root <none> and password file <none>
I20260926 03:32:33.537914 9885 fs_manager.cc:359] Metadata directory not provided
I20260926 03:32:33.538036 9885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:32:33.538374 9885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:32:33.541225 9885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-1/data/instance:
uuid: "8cd6b023e14341d29e6037a3e204c8c9"
format_stamp: "Formatted at 2026-09-26 03:32:33 on dist-test-slave-dt7p"
server_key: "c598e7cf3a618f75e25e22f9529b812b"
server_key_iv: "6ddba1f99edb62daa51f5da7a906f2a0"
server_key_version: "encryptionkey@0"
I20260926 03:32:33.542016 9885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8cd6b023e14341d29e6037a3e204c8c9"
format_stamp: "Formatted at 2026-09-26 03:32:33 on dist-test-slave-dt7p"
server_key: "c598e7cf3a618f75e25e22f9529b812b"
server_key_iv: "6ddba1f99edb62daa51f5da7a906f2a0"
server_key_version: "encryptionkey@0"
I20260926 03:32:33.547134 9885 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:32:33.550668 9899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:33.551970 9885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:32:33.552141 9885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-1/wal
uuid: "8cd6b023e14341d29e6037a3e204c8c9"
format_stamp: "Formatted at 2026-09-26 03:32:33 on dist-test-slave-dt7p"
server_key: "c598e7cf3a618f75e25e22f9529b812b"
server_key_iv: "6ddba1f99edb62daa51f5da7a906f2a0"
server_key_version: "encryptionkey@0"
I20260926 03:32:33.552345 9885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:32:33.587359 9885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:32:33.588258 9885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:32:33.588542 9885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:32:33.589541 9885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:32:33.590924 9885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:32:33.591059 9885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:33.591194 9885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:32:33.591284 9885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:33.624897 9885 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41015
I20260926 03:32:33.624940 10011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41015 every 8 connection(s)
I20260926 03:32:33.626144 9885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:32:33.628911 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 9885
I20260926 03:32:33.629099 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:32:33.629524 388 external_mini_cluster.cc:1428] Setting key efb2cde5104ba55fc87408d378b1ab01
I20260926 03:32:33.632113 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40133
--builtin_ntp_servers=127.0.97.20:41559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 03:32:33.639209 9883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:33.628713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50243 (local address 127.0.97.62:40133)
0926 03:32:33.628926 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:33.628931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:33.630535 (+ 1604us) server_negotiation.cc:407] Connection header received
0926 03:32:33.632180 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:33.632187 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:33.632273 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:33.632383 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:33.633424 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:33.634561 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:33.636178 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:33.636911 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:33.637054 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:33.638436 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:33.638445 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:33.638453 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:32:33.638489 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:33.638532 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:32:33.638551 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:33.638653 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:33.638915 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 03:32:33.638993 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:32:33.640548 10014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:33.629344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40133 (local address 127.0.97.2:50243)
0926 03:32:33.630372 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:33.630426 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:33.631918 (+ 1492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:33.632548 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:33.632556 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:33.632706 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:33.633257 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:33.633274 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:33.634740 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:33.634745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:33.636017 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:33.636032 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:33.637131 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:33.637136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:33.637347 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:33.638121 (+ 774us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:32:33.638124 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:33.638164 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:32:33.638238 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:32:33.638681 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:32:33.638694 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:32:33.638703 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 03:32:33.638905 (+ 202us) client_negotiation.cc:240] Negotiation successful
0926 03:32:33.639332 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":367,"threads_started":1}
I20260926 03:32:33.641983 10012 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40133
I20260926 03:32:33.642308 10012 heartbeater.cc:464] Registering TS with master...
I20260926 03:32:33.643199 10012 heartbeater.cc:511] Master 127.0.97.62:40133 requested a full tablet report, sending...
I20260926 03:32:33.644770 9695 ts_manager.cc:194] Registered new tserver with Master: 8cd6b023e14341d29e6037a3e204c8c9 (127.0.97.2:41015)
I20260926 03:32:33.645591 9695 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:50243
W20260926 03:32:33.967353 10015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:32:33.967617 10015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:32:33.967743 10015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:32:33.975004 10015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:32:33.975116 10015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:32:33.975272 10015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:32:33.984162 10015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:32:33.985477 10015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:32:33.987268 10015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:32:34.000221 10021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:34.000240 10023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:34.000300 10020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:32:34.000912 10015 server_base.cc:1034] running on GCE node
I20260926 03:32:34.001608 10015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:32:34.002564 10015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:32:34.003795 10015 hybrid_clock.cc:648] HybridClock initialized: now 1790393554003741 us; error 42 us; skew 500 ppm
I20260926 03:32:34.004117 10015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:32:34.006460 10015 webserver.cc:460] Webserver started at http://127.0.97.3:45389/ using document root <none> and password file <none>
I20260926 03:32:34.007158 10015 fs_manager.cc:359] Metadata directory not provided
I20260926 03:32:34.007243 10015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:32:34.007499 10015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:32:34.010300 10015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-2/data/instance:
uuid: "91c962d525894749879969ca2d232b33"
format_stamp: "Formatted at 2026-09-26 03:32:34 on dist-test-slave-dt7p"
server_key: "c2b6760054df0b4f5dcc64b5f3fe4221"
server_key_iv: "99b8a3dffd5a2ec582f87a2c654fe73d"
server_key_version: "encryptionkey@0"
I20260926 03:32:34.011049 10015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-2/wal/instance:
uuid: "91c962d525894749879969ca2d232b33"
format_stamp: "Formatted at 2026-09-26 03:32:34 on dist-test-slave-dt7p"
server_key: "c2b6760054df0b4f5dcc64b5f3fe4221"
server_key_iv: "99b8a3dffd5a2ec582f87a2c654fe73d"
server_key_version: "encryptionkey@0"
I20260926 03:32:34.016310 10015 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 03:32:34.019635 10029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:34.021059 10015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:32:34.021195 10015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-2/wal
uuid: "91c962d525894749879969ca2d232b33"
format_stamp: "Formatted at 2026-09-26 03:32:34 on dist-test-slave-dt7p"
server_key: "c2b6760054df0b4f5dcc64b5f3fe4221"
server_key_iv: "99b8a3dffd5a2ec582f87a2c654fe73d"
server_key_version: "encryptionkey@0"
I20260926 03:32:34.021366 10015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:32:34.034127 10015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:32:34.034981 10015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:32:34.035223 10015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:32:34.036087 10015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:32:34.037590 10015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:32:34.037674 10015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:34.037758 10015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:32:34.037811 10015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:34.071640 10015 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41091
I20260926 03:32:34.071686 10141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41091 every 8 connection(s)
I20260926 03:32:34.073194 10015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:32:34.078358 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 10015
I20260926 03:32:34.078540 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790393463219689-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:32:34.079074 388 external_mini_cluster.cc:1428] Setting key e89c5c2a7ef5216577e64e9fd9d4680b
I20260926 03:32:34.086911 9883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:34.075830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49953 (local address 127.0.97.62:40133)
0926 03:32:34.076043 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:34.076047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:34.077376 (+ 1329us) server_negotiation.cc:407] Connection header received
0926 03:32:34.079715 (+ 2339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:34.079721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:34.079812 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:34.079930 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:34.081287 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:34.082397 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:34.084100 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:34.084479 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:34.084623 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:34.086279 (+ 1656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:34.086284 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:34.086287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:32:34.086302 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:34.086328 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:32:34.086339 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:34.086406 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:34.086614 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 03:32:34.086683 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260926 03:32:34.087877 10144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:34.076323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40133 (local address 127.0.97.3:49953)
0926 03:32:34.077218 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:34.077258 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:34.079411 (+ 2153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:34.080163 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:34.080171 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:34.080253 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:34.081083 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:34.081106 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:34.082576 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:34.082581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:34.083963 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:34.083975 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:34.084759 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:34.084765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:34.085006 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:34.085817 (+ 811us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:32:34.085820 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:34.085862 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:32:34.086110 (+ 248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:32:34.086424 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:32:34.086432 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:32:34.086437 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 03:32:34.086578 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 03:32:34.086859 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":330,"threads_started":1}
I20260926 03:32:34.089277 10142 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40133
I20260926 03:32:34.089617 10142 heartbeater.cc:464] Registering TS with master...
I20260926 03:32:34.090404 10142 heartbeater.cc:511] Master 127.0.97.62:40133 requested a full tablet report, sending...
I20260926 03:32:34.091778 9695 ts_manager.cc:194] Registered new tserver with Master: 91c962d525894749879969ca2d232b33 (127.0.97.3:41091)
I20260926 03:32:34.092645 9695 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:49953
I20260926 03:32:34.094602 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 03:32:34.100739 9883 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:34.099071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56100 (local address 127.0.97.62:40133)
0926 03:32:34.099305 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:34.099314 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:34.099717 (+ 403us) server_negotiation.cc:407] Connection header received
0926 03:32:34.099831 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:34.099835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:34.099961 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:34.100049 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:34.100499 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56100: BlockingRecv error: recv got EOF from 127.0.0.1:56100 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 03:32:34.101302 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 9754
I20260926 03:32:34.109287 10151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:34.102203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52142 (local address 127.0.97.1:45455)
0926 03:32:34.102743 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:34.102750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:34.102773 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:32:34.102825 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:34.102832 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:34.103025 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:34.103147 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:34.103448 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:34.104429 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:34.105708 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:34.106725 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:34.106875 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:34.107882 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:34.107888 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:34.107900 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:32:34.108455 (+ 555us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:34.108497 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:32:34.108514 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:34.108694 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:34.108791 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 03:32:34.108905 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":287,"threads_started":1}
I20260926 03:32:34.110441 9876 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:32:34.192127 9881 heartbeater.cc:503] Master 127.0.97.62:40133 was elected leader, sending a full tablet report...
I20260926 03:32:34.193257 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 9754
W20260926 03:32:34.209529 9681 connection.cc:535] server connection from 127.0.97.1:34071 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 03:32:34.209892 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 9885
I20260926 03:32:34.217474 10153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:34.210711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43736 (local address 127.0.97.2:41015)
0926 03:32:34.211231 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:34.211239 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:34.211265 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:32:34.211325 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:34.211332 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:34.211573 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:34.211707 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:34.212083 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:34.213210 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:34.214394 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:34.214928 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:34.215035 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:34.215975 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:34.215983 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:34.215995 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:32:34.216756 (+ 761us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:34.216802 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:32:34.216822 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:34.216924 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:34.217063 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 03:32:34.217174 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":273,"threads_started":1}
I20260926 03:32:34.218683 10007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:32:34.299700 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 9885
I20260926 03:32:34.315488 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 10015
I20260926 03:32:34.323704 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:34.316427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37254 (local address 127.0.97.3:41091)
0926 03:32:34.317102 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:34.317109 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:34.317141 (+ 32us) server_negotiation.cc:407] Connection header received
0926 03:32:34.317206 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:34.317212 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:34.317417 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:34.317628 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:34.318071 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:34.319120 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:34.320343 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:34.321027 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:34.321169 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:34.322272 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:34.322280 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:34.322293 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:32:34.322876 (+ 583us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:34.322920 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:32:34.322952 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:34.323079 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:34.323232 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 03:32:34.323374 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":376,"threads_started":1}
I20260926 03:32:34.325136 10137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:32:34.407377 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 10015
I20260926 03:32:34.423183 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 9664
I20260926 03:32:34.424355 9725 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:32:34.526501 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 9664
2026-09-26T03:32:34Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2411 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T03:32:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:32:34Z Disabled control of system clock
I20260926 03:32:34.598423 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45213
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 03:32:34.922897 10161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:32:34.923224 10161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:32:34.923404 10161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:32:34.931164 10161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:32:34.931288 10161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:32:34.931393 10161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:32:34.931488 10161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:32:34.931550 10161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:32:34.941273 10161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45213
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:32:34.942662 10161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:32:34.944489 10161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:32:34.957258 10166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:34.957284 10167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:34.957154 10169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:32:34.957562 10161 server_base.cc:1034] running on GCE node
I20260926 03:32:34.958592 10161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:32:34.960122 10161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:32:34.961336 10161 hybrid_clock.cc:648] HybridClock initialized: now 1790393554961285 us; error 38 us; skew 500 ppm
I20260926 03:32:34.961727 10161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:32:34.964131 10161 webserver.cc:460] Webserver started at http://127.0.97.62:43397/ using document root <none> and password file <none>
I20260926 03:32:34.964905 10161 fs_manager.cc:359] Metadata directory not provided
I20260926 03:32:34.965014 10161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:32:34.965341 10161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:32:34.968215 10161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "deae0e660efd4349a1635ef6a96682d5"
format_stamp: "Formatted at 2026-09-26 03:32:34 on dist-test-slave-dt7p"
server_key: "0073b1fceef088a9c3ebb1a5ac27b1ce"
server_key_iv: "539c81204a6d2b09e11f4b0bf9980e4d"
server_key_version: "encryptionkey@0"
I20260926 03:32:34.969076 10161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "deae0e660efd4349a1635ef6a96682d5"
format_stamp: "Formatted at 2026-09-26 03:32:34 on dist-test-slave-dt7p"
server_key: "0073b1fceef088a9c3ebb1a5ac27b1ce"
server_key_iv: "539c81204a6d2b09e11f4b0bf9980e4d"
server_key_version: "encryptionkey@0"
I20260926 03:32:34.974145 10161 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:32:34.977563 10175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:34.978999 10161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:32:34.979185 10161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "deae0e660efd4349a1635ef6a96682d5"
format_stamp: "Formatted at 2026-09-26 03:32:34 on dist-test-slave-dt7p"
server_key: "0073b1fceef088a9c3ebb1a5ac27b1ce"
server_key_iv: "539c81204a6d2b09e11f4b0bf9980e4d"
server_key_version: "encryptionkey@0"
I20260926 03:32:34.979411 10161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:32:34.985328 10161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:32:34.986140 10161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:32:34.986441 10161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:32:35.005856 10161 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45213
I20260926 03:32:35.005879 10226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45213 every 8 connection(s)
I20260926 03:32:35.007288 10161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:32:35.009780 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 10161
I20260926 03:32:35.009963 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:32:35.010614 388 external_mini_cluster.cc:1428] Setting key 2a599bd6c4daa283e9c19b8f860d9be4
I20260926 03:32:35.011279 10227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:32:35.019557 10230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:35.012557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48278 (local address 127.0.97.62:45213)
0926 03:32:35.013542 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:35.013553 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:35.013597 (+ 44us) server_negotiation.cc:407] Connection header received
0926 03:32:35.015831 (+ 2234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:35.015854 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:35.016301 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:35.016784 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:35.017892 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:35.017899 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:35.017912 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:32:35.017925 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:35.017985 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:32:35.018004 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:35.018263 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:35.018416 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 03:32:35.018663 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":348,"threads_started":1}
I20260926 03:32:35.025710 10227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P deae0e660efd4349a1635ef6a96682d5: Bootstrap starting.
I20260926 03:32:35.029228 10227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P deae0e660efd4349a1635ef6a96682d5: Neither blocks nor log segments found. Creating new log.
I20260926 03:32:35.030644 10227 log.cc:824] T 00000000000000000000000000000000 P deae0e660efd4349a1635ef6a96682d5: Log is configured to *not* fsync() on all Append() calls
I20260926 03:32:35.033782 10227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P deae0e660efd4349a1635ef6a96682d5: No bootstrap required, opened a new log
I20260926 03:32:35.041600 10227 raft_consensus.cc:348] T 00000000000000000000000000000000 P deae0e660efd4349a1635ef6a96682d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deae0e660efd4349a1635ef6a96682d5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45213 } }
I20260926 03:32:35.042037 10227 raft_consensus.cc:374] T 00000000000000000000000000000000 P deae0e660efd4349a1635ef6a96682d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:32:35.042148 10227 raft_consensus.cc:729] T 00000000000000000000000000000000 P deae0e660efd4349a1635ef6a96682d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deae0e660efd4349a1635ef6a96682d5, State: Initialized, Role: FOLLOWER
I20260926 03:32:35.042791 10227 consensus_queue.cc:260] T 00000000000000000000000000000000 P deae0e660efd4349a1635ef6a96682d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deae0e660efd4349a1635ef6a96682d5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45213 } }
I20260926 03:32:35.043044 10227 raft_consensus.cc:388] T 00000000000000000000000000000000 P deae0e660efd4349a1635ef6a96682d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:32:35.043185 10227 raft_consensus.cc:482] T 00000000000000000000000000000000 P deae0e660efd4349a1635ef6a96682d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:32:35.043371 10227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P deae0e660efd4349a1635ef6a96682d5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:32:35.045668 10227 raft_consensus.cc:504] T 00000000000000000000000000000000 P deae0e660efd4349a1635ef6a96682d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deae0e660efd4349a1635ef6a96682d5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45213 } }
I20260926 03:32:35.046200 10227 leader_election.cc:302] T 00000000000000000000000000000000 P deae0e660efd4349a1635ef6a96682d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: deae0e660efd4349a1635ef6a96682d5; no voters:
I20260926 03:32:35.046768 10227 leader_election.cc:288] T 00000000000000000000000000000000 P deae0e660efd4349a1635ef6a96682d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:32:35.046969 10232 raft_consensus.cc:2781] T 00000000000000000000000000000000 P deae0e660efd4349a1635ef6a96682d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:32:35.048748 10232 raft_consensus.cc:686] T 00000000000000000000000000000000 P deae0e660efd4349a1635ef6a96682d5 [term 1 LEADER]: Becoming Leader. State: Replica: deae0e660efd4349a1635ef6a96682d5, State: Running, Role: LEADER
I20260926 03:32:35.048952 10227 sys_catalog.cc:564] T 00000000000000000000000000000000 P deae0e660efd4349a1635ef6a96682d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:32:35.049292 10232 consensus_queue.cc:237] T 00000000000000000000000000000000 P deae0e660efd4349a1635ef6a96682d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deae0e660efd4349a1635ef6a96682d5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45213 } }
I20260926 03:32:35.052181 10233 sys_catalog.cc:455] T 00000000000000000000000000000000 P deae0e660efd4349a1635ef6a96682d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "deae0e660efd4349a1635ef6a96682d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deae0e660efd4349a1635ef6a96682d5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45213 } } }
I20260926 03:32:35.052222 10234 sys_catalog.cc:455] T 00000000000000000000000000000000 P deae0e660efd4349a1635ef6a96682d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader deae0e660efd4349a1635ef6a96682d5. Latest consensus state: current_term: 1 leader_uuid: "deae0e660efd4349a1635ef6a96682d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deae0e660efd4349a1635ef6a96682d5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45213 } } }
I20260926 03:32:35.052687 10234 sys_catalog.cc:458] T 00000000000000000000000000000000 P deae0e660efd4349a1635ef6a96682d5 [sys.catalog]: This master's current role is: LEADER
I20260926 03:32:35.052712 10233 sys_catalog.cc:458] T 00000000000000000000000000000000 P deae0e660efd4349a1635ef6a96682d5 [sys.catalog]: This master's current role is: LEADER
I20260926 03:32:35.053658 10241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:32:35.060348 10241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:32:35.069777 10241 catalog_manager.cc:1347] Generated new cluster ID: d3acf0e1c8c04890aedc7d0335e8587e
I20260926 03:32:35.069913 10241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:32:35.085894 10241 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:32:35.086385 10241 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:32:35.094911 10241 catalog_manager.cc:5868] T 00000000000000000000000000000000 P deae0e660efd4349a1635ef6a96682d5: Generated new TSK 0
I20260926 03:32:35.095734 10241 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:32:35.102808 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45213
--builtin_ntp_servers=127.0.97.20:45885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 03:32:35.440843 10251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:32:35.441190 10251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:32:35.441345 10251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:32:35.449492 10251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:32:35.449611 10251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:32:35.449780 10251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:32:35.459713 10251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:32:35.461061 10251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:32:35.462792 10251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:32:35.475847 10259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:35.475945 10256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:35.475936 10257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:32:35.476891 10251 server_base.cc:1034] running on GCE node
I20260926 03:32:35.477545 10251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:32:35.478475 10251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:32:35.479676 10251 hybrid_clock.cc:648] HybridClock initialized: now 1790393555479637 us; error 37 us; skew 500 ppm
I20260926 03:32:35.479987 10251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:32:35.482204 10251 webserver.cc:460] Webserver started at http://127.0.97.1:46219/ using document root <none> and password file <none>
I20260926 03:32:35.482911 10251 fs_manager.cc:359] Metadata directory not provided
I20260926 03:32:35.483023 10251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:32:35.483345 10251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:32:35.486316 10251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-0/data/instance:
uuid: "cf7054bb52444a838dcc95eada146807"
format_stamp: "Formatted at 2026-09-26 03:32:35 on dist-test-slave-dt7p"
server_key: "2b309a0c6c723bca8cf53c0c734a0fd0"
server_key_iv: "83e1e06752854ee42e73ad397a0eedd0"
server_key_version: "encryptionkey@0"
I20260926 03:32:35.487149 10251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-0/wal/instance:
uuid: "cf7054bb52444a838dcc95eada146807"
format_stamp: "Formatted at 2026-09-26 03:32:35 on dist-test-slave-dt7p"
server_key: "2b309a0c6c723bca8cf53c0c734a0fd0"
server_key_iv: "83e1e06752854ee42e73ad397a0eedd0"
server_key_version: "encryptionkey@0"
I20260926 03:32:35.492488 10251 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 03:32:35.496100 10265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:35.497484 10251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:32:35.497665 10251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-0/wal
uuid: "cf7054bb52444a838dcc95eada146807"
format_stamp: "Formatted at 2026-09-26 03:32:35 on dist-test-slave-dt7p"
server_key: "2b309a0c6c723bca8cf53c0c734a0fd0"
server_key_iv: "83e1e06752854ee42e73ad397a0eedd0"
server_key_version: "encryptionkey@0"
I20260926 03:32:35.497901 10251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:32:35.503564 10251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:32:35.504316 10251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:32:35.504666 10251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:32:35.505633 10251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:32:35.506984 10251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:32:35.507102 10251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:35.507232 10251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:32:35.507326 10251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:35.541527 10251 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35585
I20260926 03:32:35.541580 10377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35585 every 8 connection(s)
I20260926 03:32:35.542829 10251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:32:35.546120 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 10251
I20260926 03:32:35.546311 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:32:35.546904 388 external_mini_cluster.cc:1428] Setting key 011ab026465811e0a6df1626596025fa
I20260926 03:32:35.549754 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45213
--builtin_ntp_servers=127.0.97.20:45885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 03:32:35.552429 10380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:35.545422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50417 (local address 127.0.97.62:45213)
0926 03:32:35.545943 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:35.545950 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:35.548329 (+ 2379us) server_negotiation.cc:407] Connection header received
0926 03:32:35.550164 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:35.550172 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:35.550256 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:35.550370 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:35.551690 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:35.551696 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:35.551709 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:32:35.551733 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:35.551770 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:32:35.551781 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:35.551897 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:35.552099 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 03:32:35.552161 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":242,"threads_started":1}
I20260926 03:32:35.553438 10381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:35.545989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45213 (local address 127.0.97.1:50417)
0926 03:32:35.548173 (+ 2184us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:35.548222 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:35.549883 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:35.550590 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:35.550597 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:35.550680 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:35.551393 (+ 713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:32:35.551399 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:35.551454 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:32:35.551534 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:32:35.551872 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:32:35.551881 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:32:35.551889 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 03:32:35.552174 (+ 285us) client_negotiation.cc:240] Negotiation successful
0926 03:32:35.552397 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1666,"thread_start_us":1470,"threads_started":1}
I20260926 03:32:35.554812 10378 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45213
I20260926 03:32:35.555120 10378 heartbeater.cc:464] Registering TS with master...
I20260926 03:32:35.555753 10378 heartbeater.cc:511] Master 127.0.97.62:45213 requested a full tablet report, sending...
I20260926 03:32:35.558085 10192 ts_manager.cc:194] Registered new tserver with Master: cf7054bb52444a838dcc95eada146807 (127.0.97.1:35585)
W20260926 03:32:35.870195 10382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:32:35.870589 10382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:32:35.870688 10382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:32:35.877998 10382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:32:35.878101 10382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:32:35.878207 10382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:32:35.887363 10382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:32:35.888718 10382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:32:35.890362 10382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:32:35.903332 10387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:35.903378 10388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:35.904390 10390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:32:35.906705 10382 server_base.cc:1034] running on GCE node
I20260926 03:32:35.907580 10382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:32:35.908932 10382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:32:35.910171 10382 hybrid_clock.cc:648] HybridClock initialized: now 1790393555910111 us; error 56 us; skew 500 ppm
I20260926 03:32:35.910496 10382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:32:35.912850 10382 webserver.cc:460] Webserver started at http://127.0.97.2:45119/ using document root <none> and password file <none>
I20260926 03:32:35.913558 10382 fs_manager.cc:359] Metadata directory not provided
I20260926 03:32:35.913686 10382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:32:35.913995 10382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:32:35.916900 10382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-1/data/instance:
uuid: "1267aba9a5714fa388570b77980b4995"
format_stamp: "Formatted at 2026-09-26 03:32:35 on dist-test-slave-dt7p"
server_key: "27bb1c90c745d1403ef936c0eec47126"
server_key_iv: "092fdf8087f5d54344982d8d4da5f744"
server_key_version: "encryptionkey@0"
I20260926 03:32:35.917675 10382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1267aba9a5714fa388570b77980b4995"
format_stamp: "Formatted at 2026-09-26 03:32:35 on dist-test-slave-dt7p"
server_key: "27bb1c90c745d1403ef936c0eec47126"
server_key_iv: "092fdf8087f5d54344982d8d4da5f744"
server_key_version: "encryptionkey@0"
I20260926 03:32:35.922595 10382 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:32:35.926018 10396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:35.927297 10382 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:32:35.927484 10382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-1/wal
uuid: "1267aba9a5714fa388570b77980b4995"
format_stamp: "Formatted at 2026-09-26 03:32:35 on dist-test-slave-dt7p"
server_key: "27bb1c90c745d1403ef936c0eec47126"
server_key_iv: "092fdf8087f5d54344982d8d4da5f744"
server_key_version: "encryptionkey@0"
I20260926 03:32:35.927714 10382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:32:35.933368 10382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:32:35.934141 10382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:32:35.934398 10382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:32:35.935319 10382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:32:35.936743 10382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:32:35.936856 10382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:35.936981 10382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:32:35.937084 10382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:35.970786 10382 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33813
I20260926 03:32:35.970844 10508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33813 every 8 connection(s)
I20260926 03:32:35.972056 10382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:32:35.973426 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 10382
I20260926 03:32:35.973606 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:32:35.974040 388 external_mini_cluster.cc:1428] Setting key 0d9136baed6ffb6a14d31ceac4ee5b0c
I20260926 03:32:35.976701 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45213
--builtin_ntp_servers=127.0.97.20:45885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 03:32:35.980988 10380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:35.975151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34495 (local address 127.0.97.62:45213)
0926 03:32:35.975350 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:35.975355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:35.976907 (+ 1552us) server_negotiation.cc:407] Connection header received
0926 03:32:35.978564 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:35.978572 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:35.978649 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:35.978758 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:35.980044 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:35.980053 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:35.980060 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:32:35.980091 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:35.980130 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:32:35.980147 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:35.980263 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:35.980435 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 03:32:35.980499 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 03:32:35.981837 10511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:35.975699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45213 (local address 127.0.97.2:34495)
0926 03:32:35.976762 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:35.976805 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:35.978310 (+ 1505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:35.978943 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:35.978952 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:35.979044 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:35.979755 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:32:35.979757 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:35.979800 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:32:35.979881 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:32:35.980246 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:32:35.980253 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:32:35.980258 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 03:32:35.980488 (+ 230us) client_negotiation.cc:240] Negotiation successful
0926 03:32:35.980790 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":349,"threads_started":1}
I20260926 03:32:35.983307 10509 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45213
I20260926 03:32:35.983655 10509 heartbeater.cc:464] Registering TS with master...
I20260926 03:32:35.984308 10509 heartbeater.cc:511] Master 127.0.97.62:45213 requested a full tablet report, sending...
I20260926 03:32:35.985778 10192 ts_manager.cc:194] Registered new tserver with Master: 1267aba9a5714fa388570b77980b4995 (127.0.97.2:33813)
W20260926 03:32:36.302568 10512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:32:36.302866 10512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:32:36.302955 10512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:32:36.310302 10512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:32:36.310406 10512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:32:36.310513 10512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:32:36.319645 10512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:32:36.320971 10512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:32:36.322702 10512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:32:36.335844 10518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:36.335961 10520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:36.336359 10517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:32:36.337126 10512 server_base.cc:1034] running on GCE node
I20260926 03:32:36.337687 10512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:32:36.338618 10512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:32:36.339812 10512 hybrid_clock.cc:648] HybridClock initialized: now 1790393556339779 us; error 32 us; skew 500 ppm
I20260926 03:32:36.340135 10512 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:32:36.342402 10512 webserver.cc:460] Webserver started at http://127.0.97.3:41093/ using document root <none> and password file <none>
I20260926 03:32:36.343065 10512 fs_manager.cc:359] Metadata directory not provided
I20260926 03:32:36.343149 10512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:32:36.343489 10512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:32:36.346361 10512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-2/data/instance:
uuid: "3366e7e0a54742bea163f58fe6d1b3c8"
format_stamp: "Formatted at 2026-09-26 03:32:36 on dist-test-slave-dt7p"
server_key: "3a25f85e764dfab12987f1fb85d83b52"
server_key_iv: "963183aa8ff5f0d0d5b9b6de2c2f846a"
server_key_version: "encryptionkey@0"
I20260926 03:32:36.347150 10512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-2/wal/instance:
uuid: "3366e7e0a54742bea163f58fe6d1b3c8"
format_stamp: "Formatted at 2026-09-26 03:32:36 on dist-test-slave-dt7p"
server_key: "3a25f85e764dfab12987f1fb85d83b52"
server_key_iv: "963183aa8ff5f0d0d5b9b6de2c2f846a"
server_key_version: "encryptionkey@0"
I20260926 03:32:36.352279 10512 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 03:32:36.355721 10526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:36.357163 10512 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:32:36.357329 10512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-2/wal
uuid: "3366e7e0a54742bea163f58fe6d1b3c8"
format_stamp: "Formatted at 2026-09-26 03:32:36 on dist-test-slave-dt7p"
server_key: "3a25f85e764dfab12987f1fb85d83b52"
server_key_iv: "963183aa8ff5f0d0d5b9b6de2c2f846a"
server_key_version: "encryptionkey@0"
I20260926 03:32:36.357553 10512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:32:36.363602 10512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:32:36.364426 10512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:32:36.364745 10512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:32:36.365698 10512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:32:36.367177 10512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:32:36.367266 10512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:36.367342 10512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:32:36.367395 10512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:36.400298 10512 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45333
I20260926 03:32:36.400349 10638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45333 every 8 connection(s)
I20260926 03:32:36.401818 10512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:32:36.409701 10380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:36.404684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36499 (local address 127.0.97.62:45213)
0926 03:32:36.404877 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:36.404883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:36.406151 (+ 1268us) server_negotiation.cc:407] Connection header received
0926 03:32:36.407706 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:36.407710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:36.407762 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:36.407851 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:36.409124 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:36.409128 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:36.409132 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:32:36.409146 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:36.409175 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:32:36.409186 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:36.409284 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:36.409443 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 03:32:36.409490 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 03:32:36.410040 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 10512
I20260926 03:32:36.410198 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790393463219689-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:32:36.410740 388 external_mini_cluster.cc:1428] Setting key 100fd2745c67d09b03addbd1aff21178
I20260926 03:32:36.410959 10641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:36.405050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45213 (local address 127.0.97.3:36499)
0926 03:32:36.405983 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:36.406038 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:36.407480 (+ 1442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:36.408068 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:36.408076 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:36.408158 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:36.408816 (+ 658us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:32:36.408819 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:36.408854 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:32:36.408966 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:32:36.409274 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:32:36.409281 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:32:36.409286 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 03:32:36.409493 (+ 207us) client_negotiation.cc:240] Negotiation successful
0926 03:32:36.409744 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":307,"threads_started":1}
I20260926 03:32:36.412392 10639 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45213
I20260926 03:32:36.412730 10639 heartbeater.cc:464] Registering TS with master...
I20260926 03:32:36.413281 10639 heartbeater.cc:511] Master 127.0.97.62:45213 requested a full tablet report, sending...
I20260926 03:32:36.414613 10191 ts_manager.cc:194] Registered new tserver with Master: 3366e7e0a54742bea163f58fe6d1b3c8 (127.0.97.3:45333)
I20260926 03:32:36.426116 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 03:32:36.431988 10380 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:32:36.430435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48288 (local address 127.0.97.62:45213)
0926 03:32:36.430682 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:36.430691 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:36.431101 (+ 410us) server_negotiation.cc:407] Connection header received
0926 03:32:36.431262 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:36.431265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:36.431318 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:36.431398 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:36.431764 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48288: BlockingRecv error: recv got EOF from 127.0.0.1:48288 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 03:32:36.432327 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 10251
I20260926 03:32:36.436055 10648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:36.433418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46516 (local address 127.0.97.1:35585)
0926 03:32:36.433921 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:36.433927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:36.433951 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:32:36.434006 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:36.434012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:36.434217 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:36.434377 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:36.435378 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:36.435384 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:36.435395 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:32:36.435413 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:36.435450 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:32:36.435464 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:36.435615 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:36.435690 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 03:32:36.435790 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":233,"threads_started":1}
I20260926 03:32:36.437562 10373 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:32:36.521023 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 10251
I20260926 03:32:36.537298 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 10382
I20260926 03:32:36.540717 10650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:36.538197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54660 (local address 127.0.97.2:33813)
0926 03:32:36.538728 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:36.538734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:36.538763 (+ 29us) server_negotiation.cc:407] Connection header received
0926 03:32:36.538826 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:36.538831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:36.539063 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:36.539189 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:36.540051 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:36.540057 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:36.540067 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:32:36.540103 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:36.540141 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:32:36.540155 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:36.540256 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:36.540349 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 03:32:36.540424 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":263,"threads_started":1}
I20260926 03:32:36.541926 10504 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:32:36.622810 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 10382
I20260926 03:32:36.638777 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 10512
I20260926 03:32:36.642830 10652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:36.639750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41776 (local address 127.0.97.3:45333)
0926 03:32:36.640470 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:36.640477 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:36.640507 (+ 30us) server_negotiation.cc:407] Connection header received
0926 03:32:36.640646 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:36.640653 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:36.640867 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:36.641032 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:36.642126 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:36.642132 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:36.642143 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:32:36.642158 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:36.642192 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:32:36.642204 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:36.642383 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:36.642451 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 03:32:36.642549 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":391,"threads_started":1}
I20260926 03:32:36.644199 10634 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:32:36.724052 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 10512
I20260926 03:32:36.740314 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 10161
I20260926 03:32:36.741544 10222 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:32:36.842284 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 10161
2026-09-26T03:32:36Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2313 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:32:36 dist-test-slave-dt7p krb5kdc[10657](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:32:36 dist-test-slave-dt7p krb5kdc[10657](info): set up 2 sockets
Sep 26 03:32:36 dist-test-slave-dt7p krb5kdc[10657](info): commencing operation
krb5kdc: starting...
W20260926 03:32:37.957576 388 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:32:38 dist-test-slave-dt7p krb5kdc[10657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:32:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:32:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:32:38.186291 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32797
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32797
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:32:38.505201 10670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:32:38.505522 10670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:32:38.505671 10670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:32:38.512836 10670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:32:38.513006 10670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:32:38.513111 10670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:32:38.513211 10670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:32:38.513286 10670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:32:38.522680 10670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:32:38.523979 10670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:32:38.525312 10670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:32:38.537173 10675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:38.537221 10678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:38.537262 10676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:32:38.543361 10670 server_base.cc:1034] running on GCE node
I20260926 03:32:38.544194 10670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:32:38.545804 10670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:32:38.547040 10670 hybrid_clock.cc:648] HybridClock initialized: now 1790393558547007 us; error 50 us; skew 500 ppm
I20260926 03:32:38.547400 10670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:32:38 dist-test-slave-dt7p krb5kdc[10657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:32:38.555124 10670 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:32:38.556727 10670 webserver.cc:460] Webserver started at http://127.0.97.62:32827/ using document root <none> and password file <none>
I20260926 03:32:38.557447 10670 fs_manager.cc:359] Metadata directory not provided
I20260926 03:32:38.557595 10670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:32:38.557890 10670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:32:38.560853 10670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "03b5588438f34a1998fbc78baeeb9707"
format_stamp: "Formatted at 2026-09-26 03:32:38 on dist-test-slave-dt7p"
server_key: "24977392524c512e844dab29b8750097"
server_key_iv: "790859d0cfaec02e88beca08c2a20043"
server_key_version: "encryptionkey@0"
I20260926 03:32:38.561671 10670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "03b5588438f34a1998fbc78baeeb9707"
format_stamp: "Formatted at 2026-09-26 03:32:38 on dist-test-slave-dt7p"
server_key: "24977392524c512e844dab29b8750097"
server_key_iv: "790859d0cfaec02e88beca08c2a20043"
server_key_version: "encryptionkey@0"
I20260926 03:32:38.566613 10670 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:32:38.569962 10685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:38.571211 10670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260926 03:32:38.571393 10670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "03b5588438f34a1998fbc78baeeb9707"
format_stamp: "Formatted at 2026-09-26 03:32:38 on dist-test-slave-dt7p"
server_key: "24977392524c512e844dab29b8750097"
server_key_iv: "790859d0cfaec02e88beca08c2a20043"
server_key_version: "encryptionkey@0"
I20260926 03:32:38.571589 10670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:32:38.617452 10670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:32:38.621878 10670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:32:38.622191 10670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:32:38.640786 10670 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32797
I20260926 03:32:38.640828 10736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32797 every 8 connection(s)
I20260926 03:32:38.642148 10670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:32:38.645805 10737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:32:38.647418 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 10670
I20260926 03:32:38.647617 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:32:38.648430 388 external_mini_cluster.cc:1428] Setting key 0ebd59b878667b04ae678103925f2abd
I20260926 03:32:38.660429 10737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03b5588438f34a1998fbc78baeeb9707: Bootstrap starting.
Sep 26 03:32:38 dist-test-slave-dt7p krb5kdc[10657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:32:38.664047 10737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03b5588438f34a1998fbc78baeeb9707: Neither blocks nor log segments found. Creating new log.
I20260926 03:32:38.665381 10737 log.cc:824] T 00000000000000000000000000000000 P 03b5588438f34a1998fbc78baeeb9707: Log is configured to *not* fsync() on all Append() calls
I20260926 03:32:38.668267 10737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03b5588438f34a1998fbc78baeeb9707: No bootstrap required, opened a new log
I20260926 03:32:38.671689 10740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:38.650522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38868 (local address 127.0.97.62:32797)
0926 03:32:38.651338 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:38.651350 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:38.651391 (+ 41us) server_negotiation.cc:407] Connection header received
0926 03:32:38.653030 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:38.653051 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:38.653422 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:38.653741 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:38.655501 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:38.656818 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:38.658567 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:38.659208 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:38.659330 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:38.663998 (+ 4668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:38.664006 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:38.664021 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:38.664066 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:38.667227 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:38.667923 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:38.667929 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:38.667935 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:38.668095 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:38.668499 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:38.668504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:38.668507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:38.669238 (+ 731us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:38.669523 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:38.669896 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 03:32:38.670347 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":384,"threads_started":1}
I20260926 03:32:38.677364 10737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03b5588438f34a1998fbc78baeeb9707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03b5588438f34a1998fbc78baeeb9707" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32797 } }
I20260926 03:32:38.677713 10737 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03b5588438f34a1998fbc78baeeb9707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:32:38.677785 10737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03b5588438f34a1998fbc78baeeb9707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03b5588438f34a1998fbc78baeeb9707, State: Initialized, Role: FOLLOWER
I20260926 03:32:38.678320 10737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03b5588438f34a1998fbc78baeeb9707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03b5588438f34a1998fbc78baeeb9707" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32797 } }
I20260926 03:32:38.678534 10737 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03b5588438f34a1998fbc78baeeb9707 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:32:38.678628 10737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03b5588438f34a1998fbc78baeeb9707 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:32:38.678746 10737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03b5588438f34a1998fbc78baeeb9707 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:32:38.680899 10737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03b5588438f34a1998fbc78baeeb9707 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03b5588438f34a1998fbc78baeeb9707" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32797 } }
I20260926 03:32:38.681326 10737 leader_election.cc:302] T 00000000000000000000000000000000 P 03b5588438f34a1998fbc78baeeb9707 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03b5588438f34a1998fbc78baeeb9707; no voters:
I20260926 03:32:38.681914 10737 leader_election.cc:288] T 00000000000000000000000000000000 P 03b5588438f34a1998fbc78baeeb9707 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:32:38.681998 10742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03b5588438f34a1998fbc78baeeb9707 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:32:38.682789 10742 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03b5588438f34a1998fbc78baeeb9707 [term 1 LEADER]: Becoming Leader. State: Replica: 03b5588438f34a1998fbc78baeeb9707, State: Running, Role: LEADER
I20260926 03:32:38.683435 10742 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03b5588438f34a1998fbc78baeeb9707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03b5588438f34a1998fbc78baeeb9707" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32797 } }
I20260926 03:32:38.684240 10737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03b5588438f34a1998fbc78baeeb9707 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:32:38.686564 10744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03b5588438f34a1998fbc78baeeb9707 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03b5588438f34a1998fbc78baeeb9707. Latest consensus state: current_term: 1 leader_uuid: "03b5588438f34a1998fbc78baeeb9707" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03b5588438f34a1998fbc78baeeb9707" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32797 } } }
I20260926 03:32:38.686892 10744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03b5588438f34a1998fbc78baeeb9707 [sys.catalog]: This master's current role is: LEADER
I20260926 03:32:38.686635 10743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03b5588438f34a1998fbc78baeeb9707 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03b5588438f34a1998fbc78baeeb9707" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03b5588438f34a1998fbc78baeeb9707" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32797 } } }
I20260926 03:32:38.687052 10743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03b5588438f34a1998fbc78baeeb9707 [sys.catalog]: This master's current role is: LEADER
I20260926 03:32:38.689445 10749 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:32:38.694681 10749 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:32:38.703424 10749 catalog_manager.cc:1347] Generated new cluster ID: bc36ccad973f4bd19810a1b9505f69d5
I20260926 03:32:38.703585 10749 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:32:38.715350 10749 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:32:38.716717 10749 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:32:38.726541 10749 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03b5588438f34a1998fbc78baeeb9707: Generated new TSK 0
I20260926 03:32:38.727432 10749 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:32:38.787736 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32797
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:32:39.138343 10763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:32:39.138598 10763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:32:39.138767 10763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:32:39.145782 10763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:32:39.145877 10763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:32:39.146090 10763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:32:39.155269 10763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32797
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:32:39.156522 10763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:32:39.157779 10763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:32:39.170362 10769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:39.170392 10771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:32:39.171243 10763 server_base.cc:1034] running on GCE node
W20260926 03:32:39.171418 10768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:32:39.172154 10763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:32:39.173134 10763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:32:39.174369 10763 hybrid_clock.cc:648] HybridClock initialized: now 1790393559174307 us; error 53 us; skew 500 ppm
I20260926 03:32:39.174701 10763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:32:39 dist-test-slave-dt7p krb5kdc[10657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:32:39.177820 10763 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:32:39.179284 10763 webserver.cc:460] Webserver started at http://127.0.97.1:34601/ using document root <none> and password file <none>
I20260926 03:32:39.179975 10763 fs_manager.cc:359] Metadata directory not provided
I20260926 03:32:39.180130 10763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:32:39.180423 10763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:32:39.183477 10763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-0/data/instance:
uuid: "d49d2fbf8132496abab49ead75f7d102"
format_stamp: "Formatted at 2026-09-26 03:32:39 on dist-test-slave-dt7p"
server_key: "10ef0f73878f0fd1c0fe30ba5f1694db"
server_key_iv: "770b6d223e5675393a496fdead18e07d"
server_key_version: "encryptionkey@0"
I20260926 03:32:39.184278 10763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d49d2fbf8132496abab49ead75f7d102"
format_stamp: "Formatted at 2026-09-26 03:32:39 on dist-test-slave-dt7p"
server_key: "10ef0f73878f0fd1c0fe30ba5f1694db"
server_key_iv: "770b6d223e5675393a496fdead18e07d"
server_key_version: "encryptionkey@0"
I20260926 03:32:39.189579 10763 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 03:32:39.192999 10778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:39.194178 10763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20260926 03:32:39.194353 10763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-0/wal
uuid: "d49d2fbf8132496abab49ead75f7d102"
format_stamp: "Formatted at 2026-09-26 03:32:39 on dist-test-slave-dt7p"
server_key: "10ef0f73878f0fd1c0fe30ba5f1694db"
server_key_iv: "770b6d223e5675393a496fdead18e07d"
server_key_version: "encryptionkey@0"
I20260926 03:32:39.194558 10763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:32:39.220919 10763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:32:39.225449 10763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:32:39.225768 10763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:32:39.226816 10763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:32:39.228129 10763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:32:39.228199 10763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:39.228281 10763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:32:39.228324 10763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:39.261269 10763 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41723
I20260926 03:32:39.261356 10890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41723 every 8 connection(s)
I20260926 03:32:39.262502 10763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:32:39.263034 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 10763
I20260926 03:32:39.263262 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:32:39.264036 388 external_mini_cluster.cc:1428] Setting key 3ac52559ada525fbead41a90753cbef1
Sep 26 03:32:39 dist-test-slave-dt7p krb5kdc[10657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:32:39.284045 10893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:39.265074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53603 (local address 127.0.97.62:32797)
0926 03:32:39.265577 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:39.265582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:39.266697 (+ 1115us) server_negotiation.cc:407] Connection header received
0926 03:32:39.268518 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:39.268526 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:39.268638 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:39.268753 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:39.270145 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:39.271306 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:39.273076 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:39.273452 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:39.273559 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:39.277827 (+ 4268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:39.277832 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:39.277835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:39.277866 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:39.280536 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:39.281486 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:39.281493 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:39.281499 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:39.281625 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:39.282102 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:39.282111 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:39.282114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:39.282482 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:39.282660 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:39.283394 (+ 734us) server_negotiation.cc:299] Negotiation successful
0926 03:32:39.283643 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":262,"threads_started":1}
I20260926 03:32:39.285374 10894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:39.265685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32797 (local address 127.0.97.1:53603)
0926 03:32:39.266511 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:39.266571 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:39.268236 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:39.268921 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:39.268930 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:39.269410 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:39.269971 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:39.269987 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:39.271459 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:39.271463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:39.272921 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:39.272936 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:39.273695 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:39.273706 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:39.274017 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:39.274865 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:39.274900 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:39.277605 (+ 2705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:32:39.280765 (+ 3160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:32:39.280775 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:32:39.280792 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:32:39.281333 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:32:39.281751 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:32:39.281757 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:32:39.281761 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:32:39.281975 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:32:39.282632 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:32:39.282642 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:32:39.282999 (+ 357us) client_negotiation.cc:769] Sending connection context
0926 03:32:39.283299 (+ 300us) client_negotiation.cc:240] Negotiation successful
0926 03:32:39.283850 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":352,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 03:32:39.287189 10891 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32797
I20260926 03:32:39.287585 10891 heartbeater.cc:464] Registering TS with master...
I20260926 03:32:39.288304 10891 heartbeater.cc:511] Master 127.0.97.62:32797 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 03:32:39.290530 10702 ts_manager.cc:194] Registered new tserver with Master: d49d2fbf8132496abab49ead75f7d102 (127.0.97.1:41723)
I20260926 03:32:39.291983 10702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:32:39.316210 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32797
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:32:39.633965 10897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:32:39.634215 10897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:32:39.634305 10897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:32:39.642203 10897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:32:39.642329 10897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:32:39.642488 10897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:32:39.657462 10897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32797
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:32:39.658916 10897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:32:39.660480 10897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:32:39.673765 10905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:39.673786 10903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:39.673990 10902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:32:39.675004 10897 server_base.cc:1034] running on GCE node
I20260926 03:32:39.675580 10897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:32:39.676538 10897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:32:39.677793 10897 hybrid_clock.cc:648] HybridClock initialized: now 1790393559677757 us; error 35 us; skew 500 ppm
I20260926 03:32:39.678114 10897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:32:39 dist-test-slave-dt7p krb5kdc[10657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:32:39.681249 10897 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:32:39.682730 10897 webserver.cc:460] Webserver started at http://127.0.97.2:44849/ using document root <none> and password file <none>
I20260926 03:32:39.683413 10897 fs_manager.cc:359] Metadata directory not provided
I20260926 03:32:39.683552 10897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:32:39.683842 10897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:32:39.686888 10897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-1/data/instance:
uuid: "363624c9c42a4f64903d16cb83c1e872"
format_stamp: "Formatted at 2026-09-26 03:32:39 on dist-test-slave-dt7p"
server_key: "c0420a89023287b660bfa265cfd7017e"
server_key_iv: "0b45030cb94b253c781b48093b94f5aa"
server_key_version: "encryptionkey@0"
I20260926 03:32:39.687628 10897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-1/wal/instance:
uuid: "363624c9c42a4f64903d16cb83c1e872"
format_stamp: "Formatted at 2026-09-26 03:32:39 on dist-test-slave-dt7p"
server_key: "c0420a89023287b660bfa265cfd7017e"
server_key_iv: "0b45030cb94b253c781b48093b94f5aa"
server_key_version: "encryptionkey@0"
I20260926 03:32:39.692689 10897 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 03:32:39.695842 10912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:39.697166 10897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:32:39.697321 10897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-1/wal
uuid: "363624c9c42a4f64903d16cb83c1e872"
format_stamp: "Formatted at 2026-09-26 03:32:39 on dist-test-slave-dt7p"
server_key: "c0420a89023287b660bfa265cfd7017e"
server_key_iv: "0b45030cb94b253c781b48093b94f5aa"
server_key_version: "encryptionkey@0"
I20260926 03:32:39.697506 10897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:32:39.715154 10897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:32:39.719310 10897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:32:39.719589 10897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:32:39.720546 10897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:32:39.721900 10897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:32:39.722003 10897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:39.722157 10897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:32:39.722229 10897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:39.752538 10897 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35757
I20260926 03:32:39.752635 11024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35757 every 8 connection(s)
I20260926 03:32:39.753786 10897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:32:39.756920 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 10897
I20260926 03:32:39.757133 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:32:39.757730 388 external_mini_cluster.cc:1428] Setting key ea6820a32818ad9c4a95884fe5fd2b54
Sep 26 03:32:39 dist-test-slave-dt7p krb5kdc[10657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:32:39.775945 10893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:39.756328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46965 (local address 127.0.97.62:32797)
0926 03:32:39.756502 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:39.756507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:39.757850 (+ 1343us) server_negotiation.cc:407] Connection header received
0926 03:32:39.759986 (+ 2136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:39.759990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:39.760049 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:39.760131 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:39.762159 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:39.762845 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:39.765239 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:39.765640 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:39.765796 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:39.770132 (+ 4336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:39.770136 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:39.770139 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:39.770168 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:39.772732 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:39.773616 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:39.773620 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:39.773622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:39.773730 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:39.774258 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:39.774261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:39.774263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:39.774497 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:39.774658 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:39.775453 (+ 795us) server_negotiation.cc:299] Negotiation successful
0926 03:32:39.775671 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:32:39.777350 11027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:39.756839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32797 (local address 127.0.97.2:46965)
0926 03:32:39.757683 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:39.757739 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:39.759709 (+ 1970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:39.760391 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:39.760402 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:39.761186 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:39.761961 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:39.761987 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:39.763004 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:39.763012 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:39.765104 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:39.765121 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:39.765893 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:39.765902 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:39.766231 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:39.766932 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:39.766970 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:39.769900 (+ 2930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:32:39.772873 (+ 2973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:32:39.772884 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:32:39.772901 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:32:39.773499 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:32:39.773831 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:32:39.773838 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:32:39.773844 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:32:39.774092 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:32:39.774619 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:32:39.774630 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:32:39.775061 (+ 431us) client_negotiation.cc:769] Sending connection context
0926 03:32:39.775354 (+ 293us) client_negotiation.cc:240] Negotiation successful
0926 03:32:39.775896 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":330,"threads_started":1}
I20260926 03:32:39.778878 11025 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32797
I20260926 03:32:39.779196 11025 heartbeater.cc:464] Registering TS with master...
I20260926 03:32:39.779872 11025 heartbeater.cc:511] Master 127.0.97.62:32797 requested a full tablet report, sending...
I20260926 03:32:39.781319 10702 ts_manager.cc:194] Registered new tserver with Master: 363624c9c42a4f64903d16cb83c1e872 (127.0.97.2:35757)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 03:32:39.782186 10702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:46965
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:32:39.811586 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32797
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:32:40.136196 11030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:32:40.136444 11030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:32:40.136546 11030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:32:40.143473 11030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:32:40.143566 11030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:32:40.143673 11030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:32:40.152861 11030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32797
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:32:40.154021 11030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:32:40.155195 11030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:32:40.167785 11035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:40.167757 11036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:32:40.167829 11038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:32:40.168941 11030 server_base.cc:1034] running on GCE node
I20260926 03:32:40.169590 11030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:32:40.170486 11030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:32:40.171717 11030 hybrid_clock.cc:648] HybridClock initialized: now 1790393560171685 us; error 40 us; skew 500 ppm
I20260926 03:32:40.172039 11030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:32:40 dist-test-slave-dt7p krb5kdc[10657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:32:40.175223 11030 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:32:40.176664 11030 webserver.cc:460] Webserver started at http://127.0.97.3:43921/ using document root <none> and password file <none>
I20260926 03:32:40.177280 11030 fs_manager.cc:359] Metadata directory not provided
I20260926 03:32:40.177371 11030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:32:40.177608 11030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:32:40.180289 11030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-2/data/instance:
uuid: "66cd2363888d470e9d6c57478aa0ca9d"
format_stamp: "Formatted at 2026-09-26 03:32:40 on dist-test-slave-dt7p"
server_key: "866e425c21c2ac5b146f8b573fe0bc8a"
server_key_iv: "6403edebc3ddf4c8b4f0112b5992e4ef"
server_key_version: "encryptionkey@0"
I20260926 03:32:40.181190 11030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-2/wal/instance:
uuid: "66cd2363888d470e9d6c57478aa0ca9d"
format_stamp: "Formatted at 2026-09-26 03:32:40 on dist-test-slave-dt7p"
server_key: "866e425c21c2ac5b146f8b573fe0bc8a"
server_key_iv: "6403edebc3ddf4c8b4f0112b5992e4ef"
server_key_version: "encryptionkey@0"
I20260926 03:32:40.186380 11030 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:32:40.189563 11045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:40.190804 11030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 03:32:40.190973 11030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-2/wal
uuid: "66cd2363888d470e9d6c57478aa0ca9d"
format_stamp: "Formatted at 2026-09-26 03:32:40 on dist-test-slave-dt7p"
server_key: "866e425c21c2ac5b146f8b573fe0bc8a"
server_key_iv: "6403edebc3ddf4c8b4f0112b5992e4ef"
server_key_version: "encryptionkey@0"
I20260926 03:32:40.191145 11030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:32:40.211869 11030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:32:40.215655 11030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:32:40.215941 11030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:32:40.216907 11030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:32:40.218178 11030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:32:40.218250 11030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:40.218431 11030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:32:40.218483 11030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:32:40.250741 11030 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38255
I20260926 03:32:40.250797 11157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38255 every 8 connection(s)
I20260926 03:32:40.252020 11030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:32:40.254521 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 11030
I20260926 03:32:40.254698 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:32:40.255122 388 external_mini_cluster.cc:1428] Setting key ac4468760be886713e45a17d15ca96a0
Sep 26 03:32:40 dist-test-slave-dt7p krb5kdc[10657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:32:40.272336 10893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:40.254562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53357 (local address 127.0.97.62:32797)
0926 03:32:40.254784 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:40.254791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:40.256100 (+ 1309us) server_negotiation.cc:407] Connection header received
0926 03:32:40.258484 (+ 2384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:40.258488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:40.258550 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:40.258623 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:40.260007 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:40.260903 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:40.262443 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:40.262826 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:40.262955 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:40.266386 (+ 3431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:40.266392 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:40.266398 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:40.266441 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:40.269241 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:40.270105 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:40.270112 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:40.270115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:40.270248 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:40.270714 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:40.270720 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:40.270723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:40.270975 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:40.271129 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:40.271793 (+ 664us) server_negotiation.cc:299] Negotiation successful
0926 03:32:40.271999 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 03:32:40.273298 11160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:40.255065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32797 (local address 127.0.97.3:53357)
0926 03:32:40.255910 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:40.255971 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:40.258184 (+ 2213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:40.258827 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:40.258835 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:40.259328 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:32:40.259858 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:40.259873 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:40.261045 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:40.261051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:40.262268 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:40.262305 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:40.263061 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:40.263067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:40.263278 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:40.263967 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:32:40.263993 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:32:40.266196 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:32:40.269379 (+ 3183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:32:40.269387 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:32:40.269402 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:32:40.269978 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:32:40.270360 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:32:40.270365 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:32:40.270369 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:32:40.270593 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:32:40.271106 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:32:40.271114 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:32:40.271437 (+ 323us) client_negotiation.cc:769] Sending connection context
0926 03:32:40.271694 (+ 257us) client_negotiation.cc:240] Negotiation successful
0926 03:32:40.272196 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":369,"threads_started":1}
I20260926 03:32:40.274571 11158 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32797
I20260926 03:32:40.274912 11158 heartbeater.cc:464] Registering TS with master...
I20260926 03:32:40.275739 11158 heartbeater.cc:511] Master 127.0.97.62:32797 requested a full tablet report, sending...
I20260926 03:32:40.277258 10702 ts_manager.cc:194] Registered new tserver with Master: 66cd2363888d470e9d6c57478aa0ca9d (127.0.97.3:38255)
I20260926 03:32:40.278227 10702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53357
I20260926 03:32:40.282557 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:32:40.294723 10891 heartbeater.cc:503] Master 127.0.97.62:32797 was elected leader, sending a full tablet report...
I20260926 03:32:40.299611 10893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:40.287164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38876 (local address 127.0.97.62:32797)
0926 03:32:40.287384 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:40.287393 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:40.287753 (+ 360us) server_negotiation.cc:407] Connection header received
0926 03:32:40.287883 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:40.287886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:40.287943 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:40.288022 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:40.289550 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:40.290308 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:40.291595 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:40.291979 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:40.292077 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:40.294274 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:40.294281 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:40.294285 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:40.294325 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:40.297229 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:40.297953 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:40.297956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:40.297958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:40.298060 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:40.298397 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:40.298402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:40.298405 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:40.298690 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:40.298866 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:40.299206 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 03:32:40.299368 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 03:32:40.303546 10702 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:32:40.310287 10702 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:32:40.351385 11174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:40.324607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35757 (local address 127.0.0.1:37532)
0926 03:32:40.342606 (+ 17999us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:40.342631 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:40.342754 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:40.343938 (+ 1184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:40.343945 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:40.343984 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:32:40.344419 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:40.344435 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:40.346550 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:40.346557 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:40.348615 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:40.348637 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:40.350332 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:40.350339 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:40.350536 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:40.350615 (+ 79us) client_negotiation.cc:769] Sending connection context
0926 03:32:40.350748 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 03:32:40.350942 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17825,"mutex_wait_us":101,"thread_start_us":18980,"threads_started":1}
I20260926 03:32:40.352370 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:40.325104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37532 (local address 127.0.97.2:35757)
0926 03:32:40.343255 (+ 18151us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:40.343266 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:40.343298 (+ 32us) server_negotiation.cc:407] Connection header received
0926 03:32:40.343405 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:40.343413 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:40.343695 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:40.343923 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:32:40.344674 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:40.346370 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:40.348768 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:40.350084 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:40.350244 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:40.350494 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:40.351900 (+ 1406us) server_negotiation.cc:299] Negotiation successful
0926 03:32:40.352029 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18002,"thread_start_us":17867,"threads_started":1}
I20260926 03:32:40.352749 11171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:40.327843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41723 (local address 127.0.0.1:41874)
0926 03:32:40.343540 (+ 15697us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:40.343557 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:40.343673 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:40.344290 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:40.344297 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:40.344329 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:32:40.344713 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:40.344725 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:40.346581 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:40.346587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:40.349445 (+ 2858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:40.349455 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:40.351776 (+ 2321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:40.351783 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:40.351985 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:40.352019 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 03:32:40.352116 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 03:32:40.352263 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15576,"mutex_wait_us":264,"thread_start_us":14455,"threads_started":1}
I20260926 03:32:40.353091 11175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:40.328511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41874 (local address 127.0.97.1:41723)
0926 03:32:40.333193 (+ 4682us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:40.333206 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:40.343650 (+ 10444us) server_negotiation.cc:407] Connection header received
0926 03:32:40.343839 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:40.343850 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:40.344085 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:40.344276 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:32:40.344953 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:40.346417 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:40.349669 (+ 3252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:40.351252 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:40.351390 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:40.351618 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:40.352205 (+ 587us) server_negotiation.cc:299] Negotiation successful
0926 03:32:40.352366 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4469,"thread_start_us":4273,"threads_started":1}
I20260926 03:32:40.353627 11170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:40.324368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38255 (local address 127.0.0.1:44246)
0926 03:32:40.341670 (+ 17302us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:40.341710 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:40.341892 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:40.344819 (+ 2927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:40.344823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:40.344847 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:32:40.345230 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:40.345240 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:40.346817 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:40.346821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:40.349434 (+ 2613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:40.349448 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:40.352721 (+ 3273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:40.352730 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:40.352931 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:40.352986 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 03:32:40.353078 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 03:32:40.353248 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17051,"mutex_wait_us":86,"thread_start_us":16905,"threads_started":1}
I20260926 03:32:40.353854 11172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:40.324715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44246 (local address 127.0.97.3:38255)
0926 03:32:40.344264 (+ 19549us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:40.344272 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:40.344303 (+ 31us) server_negotiation.cc:407] Connection header received
0926 03:32:40.344399 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:40.344404 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:40.344628 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:40.344826 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:32:40.345397 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:40.346654 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:40.350602 (+ 3948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:40.352179 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:40.352301 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:40.352542 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:40.353274 (+ 732us) server_negotiation.cc:299] Negotiation successful
0926 03:32:40.353462 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19362,"thread_start_us":19714,"threads_started":1}
I20260926 03:32:40.356427 10960 tablet_service.cc:1467] Processing CreateTablet for tablet 91b752ff34c1454585bd8c9c92efd5fe (DEFAULT_TABLE table=test-table [id=15ff5aa0513a4f68924f8873757f641e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:32:40.356427 10826 tablet_service.cc:1467] Processing CreateTablet for tablet 91b752ff34c1454585bd8c9c92efd5fe (DEFAULT_TABLE table=test-table [id=15ff5aa0513a4f68924f8873757f641e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:32:40.357273 11093 tablet_service.cc:1467] Processing CreateTablet for tablet 91b752ff34c1454585bd8c9c92efd5fe (DEFAULT_TABLE table=test-table [id=15ff5aa0513a4f68924f8873757f641e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:32:40.357879 10960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91b752ff34c1454585bd8c9c92efd5fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:32:40.357877 10826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91b752ff34c1454585bd8c9c92efd5fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:32:40.358379 11093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91b752ff34c1454585bd8c9c92efd5fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:32:40.371338 11176 tablet_bootstrap.cc:492] T 91b752ff34c1454585bd8c9c92efd5fe P 66cd2363888d470e9d6c57478aa0ca9d: Bootstrap starting.
I20260926 03:32:40.373350 11177 tablet_bootstrap.cc:492] T 91b752ff34c1454585bd8c9c92efd5fe P d49d2fbf8132496abab49ead75f7d102: Bootstrap starting.
I20260926 03:32:40.375429 11176 tablet_bootstrap.cc:654] T 91b752ff34c1454585bd8c9c92efd5fe P 66cd2363888d470e9d6c57478aa0ca9d: Neither blocks nor log segments found. Creating new log.
I20260926 03:32:40.376924 11177 tablet_bootstrap.cc:654] T 91b752ff34c1454585bd8c9c92efd5fe P d49d2fbf8132496abab49ead75f7d102: Neither blocks nor log segments found. Creating new log.
I20260926 03:32:40.376840 11176 log.cc:824] T 91b752ff34c1454585bd8c9c92efd5fe P 66cd2363888d470e9d6c57478aa0ca9d: Log is configured to *not* fsync() on all Append() calls
I20260926 03:32:40.378237 11177 log.cc:824] T 91b752ff34c1454585bd8c9c92efd5fe P d49d2fbf8132496abab49ead75f7d102: Log is configured to *not* fsync() on all Append() calls
I20260926 03:32:40.379660 11178 tablet_bootstrap.cc:492] T 91b752ff34c1454585bd8c9c92efd5fe P 363624c9c42a4f64903d16cb83c1e872: Bootstrap starting.
I20260926 03:32:40.380183 11176 tablet_bootstrap.cc:492] T 91b752ff34c1454585bd8c9c92efd5fe P 66cd2363888d470e9d6c57478aa0ca9d: No bootstrap required, opened a new log
I20260926 03:32:40.380457 11176 ts_tablet_manager.cc:1397] T 91b752ff34c1454585bd8c9c92efd5fe P 66cd2363888d470e9d6c57478aa0ca9d: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 03:32:40.381738 11177 tablet_bootstrap.cc:492] T 91b752ff34c1454585bd8c9c92efd5fe P d49d2fbf8132496abab49ead75f7d102: No bootstrap required, opened a new log
I20260926 03:32:40.382015 11177 ts_tablet_manager.cc:1397] T 91b752ff34c1454585bd8c9c92efd5fe P d49d2fbf8132496abab49ead75f7d102: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 03:32:40.383423 11178 tablet_bootstrap.cc:654] T 91b752ff34c1454585bd8c9c92efd5fe P 363624c9c42a4f64903d16cb83c1e872: Neither blocks nor log segments found. Creating new log.
I20260926 03:32:40.384853 11178 log.cc:824] T 91b752ff34c1454585bd8c9c92efd5fe P 363624c9c42a4f64903d16cb83c1e872: Log is configured to *not* fsync() on all Append() calls
I20260926 03:32:40.387868 11178 tablet_bootstrap.cc:492] T 91b752ff34c1454585bd8c9c92efd5fe P 363624c9c42a4f64903d16cb83c1e872: No bootstrap required, opened a new log
I20260926 03:32:40.388134 11178 ts_tablet_manager.cc:1397] T 91b752ff34c1454585bd8c9c92efd5fe P 363624c9c42a4f64903d16cb83c1e872: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260926 03:32:40.388670 11176 raft_consensus.cc:348] T 91b752ff34c1454585bd8c9c92efd5fe P 66cd2363888d470e9d6c57478aa0ca9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66cd2363888d470e9d6c57478aa0ca9d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38255 } } peers { permanent_uuid: "363624c9c42a4f64903d16cb83c1e872" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35757 } } peers { permanent_uuid: "d49d2fbf8132496abab49ead75f7d102" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41723 } }
I20260926 03:32:40.389048 11176 raft_consensus.cc:374] T 91b752ff34c1454585bd8c9c92efd5fe P 66cd2363888d470e9d6c57478aa0ca9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:32:40.389140 11176 raft_consensus.cc:729] T 91b752ff34c1454585bd8c9c92efd5fe P 66cd2363888d470e9d6c57478aa0ca9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66cd2363888d470e9d6c57478aa0ca9d, State: Initialized, Role: FOLLOWER
I20260926 03:32:40.389784 11176 consensus_queue.cc:260] T 91b752ff34c1454585bd8c9c92efd5fe P 66cd2363888d470e9d6c57478aa0ca9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66cd2363888d470e9d6c57478aa0ca9d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38255 } } peers { permanent_uuid: "363624c9c42a4f64903d16cb83c1e872" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35757 } } peers { permanent_uuid: "d49d2fbf8132496abab49ead75f7d102" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41723 } }
I20260926 03:32:40.393203 11177 raft_consensus.cc:348] T 91b752ff34c1454585bd8c9c92efd5fe P d49d2fbf8132496abab49ead75f7d102 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66cd2363888d470e9d6c57478aa0ca9d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38255 } } peers { permanent_uuid: "363624c9c42a4f64903d16cb83c1e872" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35757 } } peers { permanent_uuid: "d49d2fbf8132496abab49ead75f7d102" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41723 } }
I20260926 03:32:40.393615 11177 raft_consensus.cc:374] T 91b752ff34c1454585bd8c9c92efd5fe P d49d2fbf8132496abab49ead75f7d102 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:32:40.393721 11177 raft_consensus.cc:729] T 91b752ff34c1454585bd8c9c92efd5fe P d49d2fbf8132496abab49ead75f7d102 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d49d2fbf8132496abab49ead75f7d102, State: Initialized, Role: FOLLOWER
I20260926 03:32:40.394407 11158 heartbeater.cc:503] Master 127.0.97.62:32797 was elected leader, sending a full tablet report...
I20260926 03:32:40.394397 11177 consensus_queue.cc:260] T 91b752ff34c1454585bd8c9c92efd5fe P d49d2fbf8132496abab49ead75f7d102 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66cd2363888d470e9d6c57478aa0ca9d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38255 } } peers { permanent_uuid: "363624c9c42a4f64903d16cb83c1e872" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35757 } } peers { permanent_uuid: "d49d2fbf8132496abab49ead75f7d102" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41723 } }
I20260926 03:32:40.395115 11176 ts_tablet_manager.cc:1428] T 91b752ff34c1454585bd8c9c92efd5fe P 66cd2363888d470e9d6c57478aa0ca9d: Time spent starting tablet: real 0.015s user 0.006s sys 0.009s
I20260926 03:32:40.396071 11177 ts_tablet_manager.cc:1428] T 91b752ff34c1454585bd8c9c92efd5fe P d49d2fbf8132496abab49ead75f7d102: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260926 03:32:40.401307 11178 raft_consensus.cc:348] T 91b752ff34c1454585bd8c9c92efd5fe P 363624c9c42a4f64903d16cb83c1e872 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66cd2363888d470e9d6c57478aa0ca9d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38255 } } peers { permanent_uuid: "363624c9c42a4f64903d16cb83c1e872" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35757 } } peers { permanent_uuid: "d49d2fbf8132496abab49ead75f7d102" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41723 } }
I20260926 03:32:40.401732 11178 raft_consensus.cc:374] T 91b752ff34c1454585bd8c9c92efd5fe P 363624c9c42a4f64903d16cb83c1e872 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:32:40.401845 11178 raft_consensus.cc:729] T 91b752ff34c1454585bd8c9c92efd5fe P 363624c9c42a4f64903d16cb83c1e872 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 363624c9c42a4f64903d16cb83c1e872, State: Initialized, Role: FOLLOWER
I20260926 03:32:40.402397 11178 consensus_queue.cc:260] T 91b752ff34c1454585bd8c9c92efd5fe P 363624c9c42a4f64903d16cb83c1e872 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66cd2363888d470e9d6c57478aa0ca9d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38255 } } peers { permanent_uuid: "363624c9c42a4f64903d16cb83c1e872" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35757 } } peers { permanent_uuid: "d49d2fbf8132496abab49ead75f7d102" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41723 } }
I20260926 03:32:40.403458 11025 heartbeater.cc:503] Master 127.0.97.62:32797 was elected leader, sending a full tablet report...
I20260926 03:32:40.404052 11178 ts_tablet_manager.cc:1428] T 91b752ff34c1454585bd8c9c92efd5fe P 363624c9c42a4f64903d16cb83c1e872: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
W20260926 03:32:40.503638 11159 tablet.cc:2369] T 91b752ff34c1454585bd8c9c92efd5fe P 66cd2363888d470e9d6c57478aa0ca9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:32:40.505726 11026 tablet.cc:2369] T 91b752ff34c1454585bd8c9c92efd5fe P 363624c9c42a4f64903d16cb83c1e872: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:32:40.514809 10892 tablet.cc:2369] T 91b752ff34c1454585bd8c9c92efd5fe P d49d2fbf8132496abab49ead75f7d102: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:32:40.622123 11183 raft_consensus.cc:482] T 91b752ff34c1454585bd8c9c92efd5fe P d49d2fbf8132496abab49ead75f7d102 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:32:40.622460 11183 raft_consensus.cc:504] T 91b752ff34c1454585bd8c9c92efd5fe P d49d2fbf8132496abab49ead75f7d102 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66cd2363888d470e9d6c57478aa0ca9d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38255 } } peers { permanent_uuid: "363624c9c42a4f64903d16cb83c1e872" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35757 } } peers { permanent_uuid: "d49d2fbf8132496abab49ead75f7d102" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41723 } }
I20260926 03:32:40.624164 11183 leader_election.cc:288] T 91b752ff34c1454585bd8c9c92efd5fe P d49d2fbf8132496abab49ead75f7d102 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66cd2363888d470e9d6c57478aa0ca9d (127.0.97.3:38255), 363624c9c42a4f64903d16cb83c1e872 (127.0.97.2:35757)
I20260926 03:32:40.632966 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:40.624632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37435 (local address 127.0.97.2:35757)
0926 03:32:40.624856 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:40.624865 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:40.625524 (+ 659us) server_negotiation.cc:407] Connection header received
0926 03:32:40.625633 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:40.625637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:40.625702 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:40.625790 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:32:40.626600 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:40.627649 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:40.630605 (+ 2956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:40.631822 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:40.631992 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:40.632076 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:40.632533 (+ 457us) server_negotiation.cc:299] Negotiation successful
0926 03:32:40.632687 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 03:32:40.633020 11172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:40.624682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37465 (local address 127.0.97.3:38255)
0926 03:32:40.624912 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:40.624920 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:40.625388 (+ 468us) server_negotiation.cc:407] Connection header received
0926 03:32:40.625602 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:40.625607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:40.625686 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:40.625790 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:32:40.626479 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:40.627649 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:40.630598 (+ 2949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:40.631852 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:40.631973 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:40.632073 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:40.632658 (+ 585us) server_negotiation.cc:299] Negotiation successful
0926 03:32:40.632805 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:32:40.633121 11185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:40.624471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38255 (local address 127.0.97.1:37465)
0926 03:32:40.625249 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:40.625269 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:40.625437 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:40.625827 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:40.625832 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:40.625859 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:32:40.626321 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:40.626339 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:40.627790 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:40.627796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:40.630411 (+ 2615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:40.630419 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:40.632299 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:40.632304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:40.632431 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:40.632496 (+ 65us) client_negotiation.cc:769] Sending connection context
0926 03:32:40.632609 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 03:32:40.632724 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"mutex_wait_us":160,"thread_start_us":420,"threads_started":1}
I20260926 03:32:40.633509 11186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:40.624611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35757 (local address 127.0.97.1:37435)
0926 03:32:40.625413 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0926 03:32:40.625427 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 03:32:40.625530 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:32:40.625827 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:32:40.625832 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:32:40.625919 (+ 87us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:32:40.626415 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:40.626429 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:40.627791 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:40.627796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:40.630395 (+ 2599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:32:40.630414 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:32:40.632251 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:32:40.632256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:32:40.632374 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:40.632398 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 03:32:40.632472 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 03:32:40.632605 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"mutex_wait_us":189,"spinlock_wait_cycles":21376,"thread_start_us":465,"threads_started":1}
I20260926 03:32:40.634274 11113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b752ff34c1454585bd8c9c92efd5fe" candidate_uuid: "d49d2fbf8132496abab49ead75f7d102" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66cd2363888d470e9d6c57478aa0ca9d" is_pre_election: true
I20260926 03:32:40.634402 10980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b752ff34c1454585bd8c9c92efd5fe" candidate_uuid: "d49d2fbf8132496abab49ead75f7d102" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "363624c9c42a4f64903d16cb83c1e872" is_pre_election: true
I20260926 03:32:40.634864 11113 raft_consensus.cc:2454] T 91b752ff34c1454585bd8c9c92efd5fe P 66cd2363888d470e9d6c57478aa0ca9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d49d2fbf8132496abab49ead75f7d102 in term 0.
I20260926 03:32:40.634899 10980 raft_consensus.cc:2454] T 91b752ff34c1454585bd8c9c92efd5fe P 363624c9c42a4f64903d16cb83c1e872 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d49d2fbf8132496abab49ead75f7d102 in term 0.
I20260926 03:32:40.635555 10780 leader_election.cc:302] T 91b752ff34c1454585bd8c9c92efd5fe P d49d2fbf8132496abab49ead75f7d102 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 363624c9c42a4f64903d16cb83c1e872, 66cd2363888d470e9d6c57478aa0ca9d, d49d2fbf8132496abab49ead75f7d102; no voters:
I20260926 03:32:40.635933 11183 raft_consensus.cc:2781] T 91b752ff34c1454585bd8c9c92efd5fe P d49d2fbf8132496abab49ead75f7d102 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:32:40.636080 11183 raft_consensus.cc:482] T 91b752ff34c1454585bd8c9c92efd5fe P d49d2fbf8132496abab49ead75f7d102 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:32:40.636204 11183 raft_consensus.cc:3037] T 91b752ff34c1454585bd8c9c92efd5fe P d49d2fbf8132496abab49ead75f7d102 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:32:40.638337 11183 raft_consensus.cc:504] T 91b752ff34c1454585bd8c9c92efd5fe P d49d2fbf8132496abab49ead75f7d102 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66cd2363888d470e9d6c57478aa0ca9d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38255 } } peers { permanent_uuid: "363624c9c42a4f64903d16cb83c1e872" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35757 } } peers { permanent_uuid: "d49d2fbf8132496abab49ead75f7d102" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41723 } }
I20260926 03:32:40.639000 11183 leader_election.cc:288] T 91b752ff34c1454585bd8c9c92efd5fe P d49d2fbf8132496abab49ead75f7d102 [CANDIDATE]: Term 1 election: Requested vote from peers 66cd2363888d470e9d6c57478aa0ca9d (127.0.97.3:38255), 363624c9c42a4f64903d16cb83c1e872 (127.0.97.2:35757)
I20260926 03:32:40.639351 11113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b752ff34c1454585bd8c9c92efd5fe" candidate_uuid: "d49d2fbf8132496abab49ead75f7d102" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66cd2363888d470e9d6c57478aa0ca9d"
I20260926 03:32:40.639508 10980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b752ff34c1454585bd8c9c92efd5fe" candidate_uuid: "d49d2fbf8132496abab49ead75f7d102" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "363624c9c42a4f64903d16cb83c1e872"
I20260926 03:32:40.639568 11113 raft_consensus.cc:3037] T 91b752ff34c1454585bd8c9c92efd5fe P 66cd2363888d470e9d6c57478aa0ca9d [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:32:40.639716 10980 raft_consensus.cc:3037] T 91b752ff34c1454585bd8c9c92efd5fe P 363624c9c42a4f64903d16cb83c1e872 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:32:40.642773 11113 raft_consensus.cc:2454] T 91b752ff34c1454585bd8c9c92efd5fe P 66cd2363888d470e9d6c57478aa0ca9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d49d2fbf8132496abab49ead75f7d102 in term 1.
I20260926 03:32:40.642781 10980 raft_consensus.cc:2454] T 91b752ff34c1454585bd8c9c92efd5fe P 363624c9c42a4f64903d16cb83c1e872 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d49d2fbf8132496abab49ead75f7d102 in term 1.
I20260926 03:32:40.643190 10780 leader_election.cc:302] T 91b752ff34c1454585bd8c9c92efd5fe P d49d2fbf8132496abab49ead75f7d102 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 363624c9c42a4f64903d16cb83c1e872, d49d2fbf8132496abab49ead75f7d102; no voters:
I20260926 03:32:40.643555 11183 raft_consensus.cc:2781] T 91b752ff34c1454585bd8c9c92efd5fe P d49d2fbf8132496abab49ead75f7d102 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:32:40.644054 11183 raft_consensus.cc:686] T 91b752ff34c1454585bd8c9c92efd5fe P d49d2fbf8132496abab49ead75f7d102 [term 1 LEADER]: Becoming Leader. State: Replica: d49d2fbf8132496abab49ead75f7d102, State: Running, Role: LEADER
I20260926 03:32:40.644526 11183 consensus_queue.cc:237] T 91b752ff34c1454585bd8c9c92efd5fe P d49d2fbf8132496abab49ead75f7d102 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66cd2363888d470e9d6c57478aa0ca9d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38255 } } peers { permanent_uuid: "363624c9c42a4f64903d16cb83c1e872" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35757 } } peers { permanent_uuid: "d49d2fbf8132496abab49ead75f7d102" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41723 } }
I20260926 03:32:40.649282 10701 catalog_manager.cc:5495] T 91b752ff34c1454585bd8c9c92efd5fe P d49d2fbf8132496abab49ead75f7d102 reported cstate change: term changed from 0 to 1, leader changed from <none> to d49d2fbf8132496abab49ead75f7d102 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "d49d2fbf8132496abab49ead75f7d102" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66cd2363888d470e9d6c57478aa0ca9d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "363624c9c42a4f64903d16cb83c1e872" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d49d2fbf8132496abab49ead75f7d102" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41723 } health_report { overall_health: HEALTHY } } }
I20260926 03:32:40.713794 11175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:40.708924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41888 (local address 127.0.97.1:41723)
0926 03:32:40.709170 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:40.709174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:40.709192 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:32:40.709356 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:40.709360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:40.709440 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:40.709569 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:32:40.709862 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:40.710547 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:40.711747 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:40.712134 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:40.712219 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:40.712810 (+ 591us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:32:40.713285 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:32:40.713387 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:40.713455 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 03:32:40.713516 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 03:32:40.721419 10980 raft_consensus.cc:1261] T 91b752ff34c1454585bd8c9c92efd5fe P 363624c9c42a4f64903d16cb83c1e872 [term 1 FOLLOWER]: Refusing update from remote peer d49d2fbf8132496abab49ead75f7d102: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:32:40.721427 11113 raft_consensus.cc:1261] T 91b752ff34c1454585bd8c9c92efd5fe P 66cd2363888d470e9d6c57478aa0ca9d [term 1 FOLLOWER]: Refusing update from remote peer d49d2fbf8132496abab49ead75f7d102: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:32:40.722322 11187 consensus_queue.cc:1035] T 91b752ff34c1454585bd8c9c92efd5fe P d49d2fbf8132496abab49ead75f7d102 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66cd2363888d470e9d6c57478aa0ca9d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:32:40.722671 11183 consensus_queue.cc:1035] T 91b752ff34c1454585bd8c9c92efd5fe P d49d2fbf8132496abab49ead75f7d102 [LEADER]: Connected to new peer: Peer: permanent_uuid: "363624c9c42a4f64903d16cb83c1e872" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:32:40.747979 10702 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:32:40.748260 10702 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:32:40.751364 10702 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 03b5588438f34a1998fbc78baeeb9707: Sending DeleteTablet for 3 replicas of tablet 91b752ff34c1454585bd8c9c92efd5fe
I20260926 03:32:40.752414 11093 tablet_service.cc:1514] Processing DeleteTablet for tablet 91b752ff34c1454585bd8c9c92efd5fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:32:40 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44246
I20260926 03:32:40.752480 10826 tablet_service.cc:1514] Processing DeleteTablet for tablet 91b752ff34c1454585bd8c9c92efd5fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:32:40 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41874
I20260926 03:32:40.757354 10960 tablet_service.cc:1514] Processing DeleteTablet for tablet 91b752ff34c1454585bd8c9c92efd5fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:32:40 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37532
I20260926 03:32:40.759208 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 10763
I20260926 03:32:40.765679 11202 tablet_replica.cc:330] T 91b752ff34c1454585bd8c9c92efd5fe P d49d2fbf8132496abab49ead75f7d102: stopping tablet replica
I20260926 03:32:40.766287 11201 tablet_replica.cc:330] T 91b752ff34c1454585bd8c9c92efd5fe P 66cd2363888d470e9d6c57478aa0ca9d: stopping tablet replica
I20260926 03:32:40.766688 11201 raft_consensus.cc:2229] T 91b752ff34c1454585bd8c9c92efd5fe P 66cd2363888d470e9d6c57478aa0ca9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:32:40.767150 11201 raft_consensus.cc:2258] T 91b752ff34c1454585bd8c9c92efd5fe P 66cd2363888d470e9d6c57478aa0ca9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:32:40.767656 11202 raft_consensus.cc:2229] T 91b752ff34c1454585bd8c9c92efd5fe P d49d2fbf8132496abab49ead75f7d102 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:32:40.768250 11202 raft_consensus.cc:2258] T 91b752ff34c1454585bd8c9c92efd5fe P d49d2fbf8132496abab49ead75f7d102 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:32:40.770291 11202 ts_tablet_manager.cc:1905] T 91b752ff34c1454585bd8c9c92efd5fe P d49d2fbf8132496abab49ead75f7d102: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:32:40.774272 11201 ts_tablet_manager.cc:1905] T 91b752ff34c1454585bd8c9c92efd5fe P 66cd2363888d470e9d6c57478aa0ca9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:32:40.776089 11203 tablet_replica.cc:330] T 91b752ff34c1454585bd8c9c92efd5fe P 363624c9c42a4f64903d16cb83c1e872: stopping tablet replica
I20260926 03:32:40.776446 11203 raft_consensus.cc:2229] T 91b752ff34c1454585bd8c9c92efd5fe P 363624c9c42a4f64903d16cb83c1e872 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:32:40.776917 11203 raft_consensus.cc:2258] T 91b752ff34c1454585bd8c9c92efd5fe P 363624c9c42a4f64903d16cb83c1e872 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:32:40.782606 11203 ts_tablet_manager.cc:1905] T 91b752ff34c1454585bd8c9c92efd5fe P 363624c9c42a4f64903d16cb83c1e872: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:32:40.784675 11202 ts_tablet_manager.cc:1918] T 91b752ff34c1454585bd8c9c92efd5fe P d49d2fbf8132496abab49ead75f7d102: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:32:40.784857 11202 log.cc:1196] T 91b752ff34c1454585bd8c9c92efd5fe P d49d2fbf8132496abab49ead75f7d102: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-0/wal/wals/91b752ff34c1454585bd8c9c92efd5fe
I20260926 03:32:40.785460 11202 ts_tablet_manager.cc:1939] T 91b752ff34c1454585bd8c9c92efd5fe P d49d2fbf8132496abab49ead75f7d102: Deleting consensus metadata
Sep 26 03:32:40 dist-test-slave-dt7p krb5kdc[10657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:32:40.787142 10688 catalog_manager.cc:4841] TS d49d2fbf8132496abab49ead75f7d102 (127.0.97.1:41723): tablet 91b752ff34c1454585bd8c9c92efd5fe (table test-table [id=15ff5aa0513a4f68924f8873757f641e]) successfully deleted
I20260926 03:32:40.789023 11201 ts_tablet_manager.cc:1918] T 91b752ff34c1454585bd8c9c92efd5fe P 66cd2363888d470e9d6c57478aa0ca9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:32:40.789179 11201 log.cc:1196] T 91b752ff34c1454585bd8c9c92efd5fe P 66cd2363888d470e9d6c57478aa0ca9d: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-2/wal/wals/91b752ff34c1454585bd8c9c92efd5fe
I20260926 03:32:40.789829 11201 ts_tablet_manager.cc:1939] T 91b752ff34c1454585bd8c9c92efd5fe P 66cd2363888d470e9d6c57478aa0ca9d: Deleting consensus metadata
I20260926 03:32:40.791220 11203 ts_tablet_manager.cc:1918] T 91b752ff34c1454585bd8c9c92efd5fe P 363624c9c42a4f64903d16cb83c1e872: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:32:40.791519 11203 log.cc:1196] T 91b752ff34c1454585bd8c9c92efd5fe P 363624c9c42a4f64903d16cb83c1e872: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790393463219689-388-0/minicluster-data/ts-1/wal/wals/91b752ff34c1454585bd8c9c92efd5fe
I20260926 03:32:40.791600 10689 catalog_manager.cc:4841] TS 66cd2363888d470e9d6c57478aa0ca9d (127.0.97.3:38255): tablet 91b752ff34c1454585bd8c9c92efd5fe (table test-table [id=15ff5aa0513a4f68924f8873757f641e]) successfully deleted
I20260926 03:32:40.792058 11203 ts_tablet_manager.cc:1939] T 91b752ff34c1454585bd8c9c92efd5fe P 363624c9c42a4f64903d16cb83c1e872: Deleting consensus metadata
I20260926 03:32:40.793555 10687 catalog_manager.cc:4841] TS 363624c9c42a4f64903d16cb83c1e872 (127.0.97.2:35757): tablet 91b752ff34c1454585bd8c9c92efd5fe (table test-table [id=15ff5aa0513a4f68924f8873757f641e]) successfully deleted
I20260926 03:32:40.795701 11175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:40.761045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41892 (local address 127.0.97.1:41723)
0926 03:32:40.761219 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:40.761226 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:40.775462 (+ 14236us) server_negotiation.cc:407] Connection header received
0926 03:32:40.775645 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:40.775650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:40.775735 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:40.775868 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:40.777904 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:40.779091 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:40.781058 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:40.781777 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:40.781879 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:40.787857 (+ 5978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:40.787872 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:40.787888 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:40.787930 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:40.792074 (+ 4144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:40.792981 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:40.792990 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:40.792994 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:40.793161 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:40.793607 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:40.793612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:40.793615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:40.794128 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:40.794414 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:40.795012 (+ 598us) server_negotiation.cc:299] Negotiation successful
0926 03:32:40.795256 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 03:32:40.796365 10886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:32:40.894768 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 10763
I20260926 03:32:40.912983 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 10897
Sep 26 03:32:40 dist-test-slave-dt7p krb5kdc[10657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:32:40.928449 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:40.913731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37548 (local address 127.0.97.2:35757)
0926 03:32:40.913904 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:40.913910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:40.913956 (+ 46us) server_negotiation.cc:407] Connection header received
0926 03:32:40.914167 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:40.914172 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:40.914240 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:40.914349 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:40.915534 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:40.916236 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:40.917560 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:40.917939 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:40.918085 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:40.922457 (+ 4372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:40.922469 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:40.922486 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:40.922520 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:40.925753 (+ 3233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:40.926412 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:40.926421 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:40.926425 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:40.926579 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:40.926933 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:40.926939 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:40.926943 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:40.927313 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:40.927613 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:40.927894 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 03:32:40.928102 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 03:32:40.929137 11020 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:32:41.017097 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 10897
I20260926 03:32:41.034528 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 11030
Sep 26 03:32:41 dist-test-slave-dt7p krb5kdc[10657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:32:41.049101 11172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:32:41.035226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44262 (local address 127.0.97.3:38255)
0926 03:32:41.035402 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 03:32:41.035408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:32:41.035429 (+ 21us) server_negotiation.cc:407] Connection header received
0926 03:32:41.035572 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:32:41.035575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:32:41.035641 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:32:41.035756 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:32:41.037071 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:41.037776 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:41.038968 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:32:41.039356 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:32:41.039483 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:32:41.043655 (+ 4172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:32:41.043667 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:32:41.043683 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:32:41.043715 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:32:41.046450 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:41.047136 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:41.047144 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:41.047146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:41.047274 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:32:41.047624 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:32:41.047630 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:32:41.047633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:32:41.047966 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:32:41.048256 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0926 03:32:41.048589 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 03:32:41.048803 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 03:32:41.049763 11153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:32:41.137912 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 11030
I20260926 03:32:41.156148 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 10670
I20260926 03:32:41.157336 10732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:32:41.264415 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 10670
2026-09-26T03:32:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4423 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:32:41 dist-test-slave-dt7p krb5kdc[11211](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:32:41 dist-test-slave-dt7p krb5kdc[11211](info): set up 2 sockets
Sep 26 03:32:41 dist-test-slave-dt7p krb5kdc[11211](info): commencing operation
krb5kdc: starting...
W20260926 03:32:42.385502 388 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.004s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:32:42 dist-test-slave-dt7p krb5kdc[11211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:32:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:32:42Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260926 03:32:42.703347 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskssCcLE/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/postgres -l logfile start
2026-09-26 03:32:45.188 UTC [11241] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 03:32:45.188 UTC [11241] LOG: listening on IPv4 address "127.0.97.20", port 40703
2026-09-26 03:32:45.192 UTC [11241] LOG: listening on Unix socket "/tmp/.s.PGSQL.40703"
2026-09-26 03:32:45.215 UTC [11244] LOG: database system was shut down at 2026-09-26 03:32:43 UTC
2026-09-26 03:32:45.218 UTC [11241] LOG: database system is ready to accept connections
I20260926 03:32:46.201290 388 mini_postgres.cc:96] Postgres bound to 40703
2026-09-26 03:32:46.219 UTC [11252] FATAL: database "slave" does not exist
127.0.97.20:40703 - accepting connections
I20260926 03:32:46.219939 388 mini_ranger.cc:161] Starting Ranger...
I20260926 03:32:46.245740 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 03:32:46.739809 388 mini_ranger.cc:87] Created ranger Postgres database
I20260926 03:32:46.739940 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ranger-admin
2026-09-26 03:32:47,019 [I] DB FLAVOR :POSTGRES
2026-09-26 03:32:47,020 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 03:32:47,020 [I] Checking connection..
2026-09-26 03:32:47,020 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 03:32:47,274 [I] Checking connection passed.
2026-09-26 03:32:47,274 [I] --------- Verifying version history table ---------
2026-09-26 03:32:47,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 03:32:47,531 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 03:32:47,532 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 03:32:47,787 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 03:32:47,787 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 03:32:47,787 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40703/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 03:32:48.039 UTC [11316] WARNING: there is no transaction in progress
2026-09-26 03:32:48,045 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 03:32:48,046 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 03:32:48,306 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 03:32:48,306 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 03:32:48,306 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 03:32:48,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 03:32:48,803 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-dt7p.c.gcp-upstream.internal','N') ;"
2026-09-26 03:32:49,035 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 03:32:49,298 [I] Table x_portal_user does not exist in database ranger
2026-09-26 03:32:49,299 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 03:32:49,557 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 03:32:49,557 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 03:32:49,800 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 03:32:50,053 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 03:32:50,054 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40703/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 03:32:51.014 UTC [11424] WARNING: there is no transaction in progress
2026-09-26 03:32:51.030 UTC [11424] WARNING: there is no transaction in progress
2026-09-26 03:32:51.048 UTC [11424] WARNING: there is no transaction in progress
2026-09-26 03:32:51.063 UTC [11424] WARNING: there is no transaction in progress
2026-09-26 03:32:51.079 UTC [11424] WARNING: there is no transaction in progress
2026-09-26 03:32:51.096 UTC [11424] WARNING: there is no transaction in progress
2026-09-26 03:32:51.153 UTC [11424] WARNING: there is no transaction in progress
2026-09-26 03:32:51.160 UTC [11424] WARNING: there is no transaction in progress
2026-09-26 03:32:51.175 UTC [11424] WARNING: there is no transaction in progress
2026-09-26 03:32:51.185 UTC [11424] WARNING: there is no transaction in progress
2026-09-26 03:32:51.195 UTC [11424] WARNING: there is no transaction in progress
2026-09-26 03:32:51.209 UTC [11424] WARNING: there is no transaction in progress
2026-09-26 03:32:51.219 UTC [11424] WARNING: there is no transaction in progress
2026-09-26 03:32:51.226 UTC [11424] WARNING: there is no transaction in progress
2026-09-26 03:32:51.234 UTC [11424] WARNING: there is no transaction in progress
2026-09-26 03:32:51.241 UTC [11424] WARNING: there is no transaction in progress
2026-09-26 03:32:51,869 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 03:32:51,870 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 03:32:52,114 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 03:32:52,115 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 03:32:52,379 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 03:32:52,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 03:32:52,644 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 03:32:52,645 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 03:32:52,912 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 03:32:53,174 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-dt7p.c.gcp-upstream.internal';"
2026-09-26 03:32:53,455 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 03:32:53,456 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 03:32:53,757 [I] DB_PATCHES have already been applied
I20260926 03:32:53.763975 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskssCcLE/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 03:32:53.764173 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260926 03:32:53.784300 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:33333
Sep 26, 2026 3:32:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 3:32:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 3:32:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 3:32:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 3:32:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 3:32:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 3:32:54 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 03:32:54.976027 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.756s user 0.010s sys 0.037s
Sep 26, 2026 3:33:16 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 3:33:16 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 26, 2026 3:33:16 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 3:33:17 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 3:33:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 3:33:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 3:33:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 3:33:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 3:33:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 3:33:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 3:33:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 3:33:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 3:33:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 3:33:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 3:33:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 3:33:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 3:33:17 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 3:33:18 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 03:33:18.632992 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 03:33:18 dist-test-slave-dt7p krb5kdc[11211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393598, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 03:33:18.831887 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 03:33:18.856971 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 03:33:19.862946 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 03:33:19.863114 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ranger-kms
I20260926 03:33:19.863287 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:40703
2026-09-26 03:33:20,365 [I] DB FLAVOR :POSTGRES
2026-09-26 03:33:20,365 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 03:33:20,366 [I] Checking connection
2026-09-26 03:33:20,366 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40703/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 03:33:20,659 [I] connection success
2026-09-26 03:33:20,659 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 03:33:20,659 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 03:33:20,659 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40703/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 03:33:20,940 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 03:33:20,941 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 03:33:20,941 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 03:33:20,941 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40703/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 03:33:21,253 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 03:33:21.783053 388 mini_ranger_kms.cc:310] Created kms service
I20260926 03:33:21.990098 388 mini_ranger_kms.cc:326] Created kudu user
I20260926 03:33:22.043931 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 03:33:22 dist-test-slave-dt7p krb5kdc[11211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393598, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:33:22.553169 388 mini_ranger_kms.cc:384] Added ranger policy
I20260926 03:33:22.553568 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 03:33:22.553628 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260926 03:33:22.579489 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11605
I20260926 03:33:22.579757 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:57063
Sep 26, 2026 3:33:23 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 3:33:23 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 3:33:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 3:33:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 3:33:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 3:33:24 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 03:33:25.020787 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.189s user 0.165s sys 0.371s
I20260926 03:33:25.021092 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 03:33:25.021343 388 mini_ranger_kms.cc:401] 127.0.97.20:57063/kms/v1/keys
Sep 26 03:33:25 dist-test-slave-dt7p krb5kdc[11211](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790393605, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 03:33:25 dist-test-slave-dt7p krb5kdc[11211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393605, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 26 03:33:26 dist-test-slave-dt7p krb5kdc[11211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393605, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 26 03:33:28 dist-test-slave-dt7p krb5kdc[11211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:33:28.749209 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35133
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35133
--ranger_config_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:57063/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:33:29.173379 11655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:33:29.173724 11655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:33:29.173890 11655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:33:29.181622 11655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:33:29.181766 11655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:33:29.181856 11655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:33:29.181980 11655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:33:29.182067 11655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:33:29.194015 11655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:57063/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35133
--ranger_config_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:33:29.195691 11655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:33:29.197362 11655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:33:29.212139 11661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:33:29.212260 11660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:33:29.212921 11663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:33:29.212934 11655 server_base.cc:1034] running on GCE node
I20260926 03:33:29.213923 11655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:33:29.215689 11655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:33:29.216919 11655 hybrid_clock.cc:648] HybridClock initialized: now 1790393609216880 us; error 36 us; skew 500 ppm
I20260926 03:33:29.217327 11655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:33:29 dist-test-slave-dt7p krb5kdc[11211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:33:29.227026 11655 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:33:29.229043 11655 webserver.cc:460] Webserver started at http://127.0.97.62:32879/ using document root <none> and password file <none>
I20260926 03:33:29.229873 11655 fs_manager.cc:359] Metadata directory not provided
I20260926 03:33:29.229990 11655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:33:29.230309 11655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:33:29 dist-test-slave-dt7p krb5kdc[11211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:33:29.303124 11655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "6591e6134b054f86900ecf36692f079e"
format_stamp: "Formatted at 2026-09-26 03:33:29 on dist-test-slave-dt7p"
server_key: "c780f14dc6283cbdcef9e2bae0ce4356"
server_key_iv: "9e707dd7f6ffc43532247ca7a62e97f3"
server_key_version: "kuduclusterkey@0"
I20260926 03:33:29.304165 11655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "6591e6134b054f86900ecf36692f079e"
format_stamp: "Formatted at 2026-09-26 03:33:29 on dist-test-slave-dt7p"
server_key: "c780f14dc6283cbdcef9e2bae0ce4356"
server_key_iv: "9e707dd7f6ffc43532247ca7a62e97f3"
server_key_version: "kuduclusterkey@0"
I20260926 03:33:29.311023 11655 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260926 03:33:29.336601 11670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:33:29.338640 11655 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260926 03:33:29.338907 11655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "6591e6134b054f86900ecf36692f079e"
format_stamp: "Formatted at 2026-09-26 03:33:29 on dist-test-slave-dt7p"
server_key: "c780f14dc6283cbdcef9e2bae0ce4356"
server_key_iv: "9e707dd7f6ffc43532247ca7a62e97f3"
server_key_version: "kuduclusterkey@0"
I20260926 03:33:29.339190 11655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:33:29.357676 11655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:33:29.362844 11655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:33:29.363214 11655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:33:29.389660 11655 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35133
I20260926 03:33:29.389771 11721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35133 every 8 connection(s)
I20260926 03:33:29.391451 11655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:33:29.396136 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 11655
I20260926 03:33:29.396378 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:33:29.396384 11722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:33:29.397305 388 external_mini_cluster.cc:1428] Setting key edaadb67ec021697e4d3c890cae4697c
I20260926 03:33:29.413807 11722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6591e6134b054f86900ecf36692f079e: Bootstrap starting.
Sep 26 03:33:29 dist-test-slave-dt7p krb5kdc[11211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:33:29.418655 11722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6591e6134b054f86900ecf36692f079e: Neither blocks nor log segments found. Creating new log.
I20260926 03:33:29.420238 11722 log.cc:824] T 00000000000000000000000000000000 P 6591e6134b054f86900ecf36692f079e: Log is configured to *not* fsync() on all Append() calls
I20260926 03:33:29.424347 11722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6591e6134b054f86900ecf36692f079e: No bootstrap required, opened a new log
I20260926 03:33:29.425279 11725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:33:29.400609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44056 (local address 127.0.97.62:35133)
0926 03:33:29.401545 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0926 03:33:29.401563 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 03:33:29.401624 (+ 61us) server_negotiation.cc:407] Connection header received
0926 03:33:29.403690 (+ 2066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:33:29.403724 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:33:29.404359 (+ 635us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:33:29.404964 (+ 605us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:33:29.406471 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:33:29.408724 (+ 2253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:33:29.410112 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:33:29.411380 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:33:29.411596 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:33:29.415776 (+ 4180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:33:29.415795 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:33:29.415821 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:33:29.415894 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:33:29.419955 (+ 4061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:33:29.420763 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:33:29.420774 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:33:29.420786 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:33:29.421010 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:33:29.421492 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:33:29.421496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:33:29.421499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:33:29.422352 (+ 853us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:33:29.422668 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 03:33:29.423112 (+ 444us) server_negotiation.cc:299] Negotiation successful
0926 03:33:29.423624 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":429,"threads_started":1}
I20260926 03:33:29.434904 11722 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6591e6134b054f86900ecf36692f079e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6591e6134b054f86900ecf36692f079e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35133 } }
I20260926 03:33:29.435356 11722 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6591e6134b054f86900ecf36692f079e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:33:29.435442 11722 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6591e6134b054f86900ecf36692f079e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6591e6134b054f86900ecf36692f079e, State: Initialized, Role: FOLLOWER
I20260926 03:33:29.436072 11722 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6591e6134b054f86900ecf36692f079e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6591e6134b054f86900ecf36692f079e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35133 } }
I20260926 03:33:29.436479 11722 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6591e6134b054f86900ecf36692f079e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:33:29.436719 11722 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6591e6134b054f86900ecf36692f079e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:33:29.436923 11722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6591e6134b054f86900ecf36692f079e [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:33:29.439963 11722 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6591e6134b054f86900ecf36692f079e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6591e6134b054f86900ecf36692f079e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35133 } }
I20260926 03:33:29.440678 11722 leader_election.cc:302] T 00000000000000000000000000000000 P 6591e6134b054f86900ecf36692f079e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6591e6134b054f86900ecf36692f079e; no voters:
I20260926 03:33:29.441380 11722 leader_election.cc:288] T 00000000000000000000000000000000 P 6591e6134b054f86900ecf36692f079e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:33:29.441538 11727 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6591e6134b054f86900ecf36692f079e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:33:29.442348 11727 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6591e6134b054f86900ecf36692f079e [term 1 LEADER]: Becoming Leader. State: Replica: 6591e6134b054f86900ecf36692f079e, State: Running, Role: LEADER
I20260926 03:33:29.443081 11727 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6591e6134b054f86900ecf36692f079e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6591e6134b054f86900ecf36692f079e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35133 } }
I20260926 03:33:29.444077 11722 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6591e6134b054f86900ecf36692f079e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:33:29.446792 11728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6591e6134b054f86900ecf36692f079e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6591e6134b054f86900ecf36692f079e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6591e6134b054f86900ecf36692f079e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35133 } } }
I20260926 03:33:29.447197 11728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6591e6134b054f86900ecf36692f079e [sys.catalog]: This master's current role is: LEADER
I20260926 03:33:29.448119 11729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6591e6134b054f86900ecf36692f079e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6591e6134b054f86900ecf36692f079e. Latest consensus state: current_term: 1 leader_uuid: "6591e6134b054f86900ecf36692f079e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6591e6134b054f86900ecf36692f079e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35133 } } }
I20260926 03:33:29.448655 11729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6591e6134b054f86900ecf36692f079e [sys.catalog]: This master's current role is: LEADER
I20260926 03:33:29.452095 11732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:33:29.459362 11732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:33:29.460347 11722 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 03:33:29.484730 11732 catalog_manager.cc:1347] Generated new cluster ID: 2d87ed497af24d6c95dbd7ba7f4d795a
I20260926 03:33:29.484939 11732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:33:29.500018 11732 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:33:29.502414 11732 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:33:29.514537 11732 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6591e6134b054f86900ecf36692f079e: Generated new TSK 0
I20260926 03:33:29.516644 11732 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:33:30.361315 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:57063/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35133
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:33:30 dist-test-slave-dt7p krb5kdc[11211](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790393610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 03:33:30.882566 11767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:33:30.883056 11767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:33:30.883261 11767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:33:30.896133 11767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:33:30.896359 11767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:33:30.896631 11767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:33:30.912357 11767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:57063/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:33:30.914480 11767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:33:30.916509 11767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:33:30.936005 11778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:33:30.936340 11777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:33:30.943792 11767 server_base.cc:1034] running on GCE node
W20260926 03:33:30.944847 11780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:33:30.945717 11767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:33:30.946934 11767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:33:30.948180 11767 hybrid_clock.cc:648] HybridClock initialized: now 1790393610948111 us; error 61 us; skew 500 ppm
I20260926 03:33:30.948554 11767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:33:30 dist-test-slave-dt7p krb5kdc[11211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:33:30.952739 11767 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:33:30.955088 11767 webserver.cc:460] Webserver started at http://127.0.97.1:35975/ using document root <none> and password file <none>
I20260926 03:33:30.956187 11767 fs_manager.cc:359] Metadata directory not provided
I20260926 03:33:30.956353 11767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:33:30.956933 11767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:33:30 dist-test-slave-dt7p krb5kdc[11211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:33:30.996948 11767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-0/data/instance:
uuid: "d16cd7345a254adc949279844cf52b3b"
format_stamp: "Formatted at 2026-09-26 03:33:30 on dist-test-slave-dt7p"
server_key: "962baaded5a9f451b5f32c270f6823fd"
server_key_iv: "805684daff198c607e4c884a6d93d084"
server_key_version: "kuduclusterkey@0"
I20260926 03:33:30.999706 11767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d16cd7345a254adc949279844cf52b3b"
format_stamp: "Formatted at 2026-09-26 03:33:30 on dist-test-slave-dt7p"
server_key: "962baaded5a9f451b5f32c270f6823fd"
server_key_iv: "805684daff198c607e4c884a6d93d084"
server_key_version: "kuduclusterkey@0"
I20260926 03:33:31.023345 11767 fs_manager.cc:691] Time spent creating directory manager: real 0.023s user 0.010s sys 0.011s
I20260926 03:33:31.057557 11787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:33:31.062500 11767 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.008s sys 0.002s
I20260926 03:33:31.062898 11767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-0/wal
uuid: "d16cd7345a254adc949279844cf52b3b"
format_stamp: "Formatted at 2026-09-26 03:33:30 on dist-test-slave-dt7p"
server_key: "962baaded5a9f451b5f32c270f6823fd"
server_key_iv: "805684daff198c607e4c884a6d93d084"
server_key_version: "kuduclusterkey@0"
I20260926 03:33:31.063371 11767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:33:31.097126 11767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:33:31.103662 11767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:33:31.104048 11767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:33:31.105417 11767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:33:31.111887 11767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:33:31.112004 11767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:33:31.112156 11767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:33:31.112237 11767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:33:31 dist-test-slave-dt7p krb5kdc[11211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:33:31.249698 11767 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33305
I20260926 03:33:31.249765 11899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33305 every 8 connection(s)
I20260926 03:33:31.251853 11767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:33:31.253453 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 11767
I20260926 03:33:31.253684 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:33:31.254531 388 external_mini_cluster.cc:1428] Setting key bc0180f4ff83de7b9fd9060d254209d7
Sep 26 03:33:31 dist-test-slave-dt7p krb5kdc[11211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:33:31.289296 11903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:33:31.257197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35133 (local address 127.0.97.1:60009)
0926 03:33:31.261709 (+ 4512us) negotiation.cc:107] Waiting for socket to connect
0926 03:33:31.261772 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 03:33:31.264398 (+ 2626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:33:31.265934 (+ 1536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:33:31.265948 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:33:31.266687 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:33:31.267627 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:33:31.267654 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:33:31.269495 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:33:31.269501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:33:31.272668 (+ 3167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:33:31.272688 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:33:31.274152 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:33:31.274161 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:33:31.274520 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:33:31.275504 (+ 984us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:33:31.275540 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:33:31.278793 (+ 3253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:33:31.283385 (+ 4592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:33:31.283397 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:33:31.283422 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:33:31.284134 (+ 712us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:33:31.284656 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:33:31.284663 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:33:31.284667 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:33:31.285016 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:33:31.285846 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:33:31.285855 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:33:31.286392 (+ 537us) client_negotiation.cc:769] Sending connection context
0926 03:33:31.286794 (+ 402us) client_negotiation.cc:240] Negotiation successful
0926 03:33:31.287469 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4142,"thread_start_us":7190,"threads_started":1}
I20260926 03:33:31.290652 11902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:33:31.256324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60009 (local address 127.0.97.62:35133)
0926 03:33:31.265254 (+ 8930us) server_negotiation.cc:206] Beginning negotiation
0926 03:33:31.265261 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:33:31.265280 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:33:31.265358 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:33:31.265364 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:33:31.265454 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:33:31.265572 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:33:31.267888 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:33:31.269275 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:33:31.272890 (+ 3615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:33:31.273640 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:33:31.273790 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:33:31.279169 (+ 5379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:33:31.279179 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:33:31.279185 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:33:31.279249 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:33:31.283173 (+ 3924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:33:31.284308 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:33:31.284315 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:33:31.284319 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:33:31.284499 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:33:31.285167 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:33:31.285173 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:33:31.285179 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:33:31.285602 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:33:31.285758 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 03:33:31.289987 (+ 4229us) server_negotiation.cc:299] Negotiation successful
0926 03:33:31.290260 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8791,"thread_start_us":8637,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 03:33:31.293452 11900 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35133
I20260926 03:33:31.293857 11900 heartbeater.cc:464] Registering TS with master...
I20260926 03:33:31.294931 11900 heartbeater.cc:511] Master 127.0.97.62:35133 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 03:33:31.307237 11687 ts_manager.cc:194] Registered new tserver with Master: d16cd7345a254adc949279844cf52b3b (127.0.97.1:33305)
I20260926 03:33:31.309702 11687 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:60009
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:33:31.337476 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:57063/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35133
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:33:31 dist-test-slave-dt7p krb5kdc[11211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260926 03:33:31.863098 11906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:33:31.863505 11906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:33:31.863672 11906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:33:31.876745 11906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:33:31.877034 11906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:33:31.877288 11906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:33:31.891820 11906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:57063/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:33:31.894025 11906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:33:31.895697 11906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:33:31.919548 11920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:33:31.920091 11921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:33:31.926048 11923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:33:31.926934 11906 server_base.cc:1034] running on GCE node
I20260926 03:33:31.927753 11906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:33:31.929100 11906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:33:31.930384 11906 hybrid_clock.cc:648] HybridClock initialized: now 1790393611930289 us; error 69 us; skew 500 ppm
I20260926 03:33:31.930866 11906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:33:31 dist-test-slave-dt7p krb5kdc[11211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:33:31.934859 11906 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:33:31.936918 11906 webserver.cc:460] Webserver started at http://127.0.97.2:43293/ using document root <none> and password file <none>
I20260926 03:33:31.937804 11906 fs_manager.cc:359] Metadata directory not provided
I20260926 03:33:31.937935 11906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:33:31.938297 11906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:33:31 dist-test-slave-dt7p krb5kdc[11211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:33:31.956998 11906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-1/data/instance:
uuid: "88ae8c01e5a245f9a1d413ce104fc8fc"
format_stamp: "Formatted at 2026-09-26 03:33:31 on dist-test-slave-dt7p"
server_key: "c65f5a8d8733640fbb0d3967bf7d65b1"
server_key_iv: "4be627b85797568a0d6a2d9429e5e38d"
server_key_version: "kuduclusterkey@0"
I20260926 03:33:31.958069 11906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-1/wal/instance:
uuid: "88ae8c01e5a245f9a1d413ce104fc8fc"
format_stamp: "Formatted at 2026-09-26 03:33:31 on dist-test-slave-dt7p"
server_key: "c65f5a8d8733640fbb0d3967bf7d65b1"
server_key_iv: "4be627b85797568a0d6a2d9429e5e38d"
server_key_version: "kuduclusterkey@0"
I20260926 03:33:31.965461 11906 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260926 03:33:31.987275 11930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:33:31.989766 11906 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.004s
I20260926 03:33:31.990185 11906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-1/wal
uuid: "88ae8c01e5a245f9a1d413ce104fc8fc"
format_stamp: "Formatted at 2026-09-26 03:33:31 on dist-test-slave-dt7p"
server_key: "c65f5a8d8733640fbb0d3967bf7d65b1"
server_key_iv: "4be627b85797568a0d6a2d9429e5e38d"
server_key_version: "kuduclusterkey@0"
I20260926 03:33:31.990520 11906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:33:32.010154 11906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:33:32.016109 11906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:33:32.016639 11906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:33:32.018007 11906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:33:32.020164 11906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:33:32.020268 11906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:33:32.020406 11906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:33:32.020471 11906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:33:32.064137 11906 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33713
I20260926 03:33:32.064145 12042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33713 every 8 connection(s)
I20260926 03:33:32.066260 11906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:33:32.071882 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 11906
I20260926 03:33:32.072099 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:33:32.072801 388 external_mini_cluster.cc:1428] Setting key ec7570a7ad194e259127134d95574f9b
Sep 26 03:33:32 dist-test-slave-dt7p krb5kdc[11211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:33:32.094574 12045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:33:32.069841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57331 (local address 127.0.97.62:35133)
0926 03:33:32.070328 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0926 03:33:32.070333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:33:32.073512 (+ 3179us) server_negotiation.cc:407] Connection header received
0926 03:33:32.074305 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:33:32.074311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:33:32.074432 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:33:32.074559 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:33:32.076669 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:33:32.077957 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:33:32.080694 (+ 2737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:33:32.081403 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:33:32.081547 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:33:32.086484 (+ 4937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:33:32.086492 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:33:32.086498 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:33:32.086559 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:33:32.090394 (+ 3835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:33:32.091430 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:33:32.091437 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:33:32.091441 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:33:32.091607 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:33:32.092144 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:33:32.092149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:33:32.092154 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:33:32.092629 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:33:32.092801 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 03:33:32.093846 (+ 1045us) server_negotiation.cc:299] Negotiation successful
0926 03:33:32.094118 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":236,"threads_started":1}
I20260926 03:33:32.096125 12046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:33:32.070484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35133 (local address 127.0.97.2:57331)
0926 03:33:32.071379 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0926 03:33:32.071474 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0926 03:33:32.073972 (+ 2498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:33:32.074818 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:33:32.074828 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:33:32.075475 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:33:32.076416 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:33:32.076445 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:33:32.078148 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:33:32.078158 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:33:32.080511 (+ 2353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:33:32.080531 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:33:32.081698 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:33:32.081708 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:33:32.082087 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:33:32.083064 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:33:32.083103 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:33:32.086205 (+ 3102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:33:32.090562 (+ 4357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:33:32.090575 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:33:32.090600 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:33:32.091282 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:33:32.091733 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:33:32.091740 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:33:32.091748 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:33:32.092018 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:33:32.092775 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:33:32.092788 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:33:32.093330 (+ 542us) client_negotiation.cc:769] Sending connection context
0926 03:33:32.093689 (+ 359us) client_negotiation.cc:240] Negotiation successful
0926 03:33:32.094348 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":394,"threads_started":1}
I20260926 03:33:32.098491 12043 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35133
I20260926 03:33:32.098906 12043 heartbeater.cc:464] Registering TS with master...
I20260926 03:33:32.100839 12043 heartbeater.cc:511] Master 127.0.97.62:35133 requested a full tablet report, sending...
I20260926 03:33:32.102926 11687 ts_manager.cc:194] Registered new tserver with Master: 88ae8c01e5a245f9a1d413ce104fc8fc (127.0.97.2:33713)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 03:33:32.104172 11687 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57331
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:33:32.143780 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:57063/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35133
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:33:32.314648 11900 heartbeater.cc:503] Master 127.0.97.62:35133 was elected leader, sending a full tablet report...
W20260926 03:33:32.579223 12049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:33:32.579574 12049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:33:32.579728 12049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:33:32.587612 12049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:33:32.587733 12049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:33:32.587950 12049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:33:32.598826 12049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:57063/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:33:32.600437 12049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:33:32.602734 12049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:33:32.616789 12055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:33:32.616856 12054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:33:32.618263 12057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:33:32.619076 12049 server_base.cc:1034] running on GCE node
I20260926 03:33:32.619756 12049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:33:32.620883 12049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:33:32.622110 12049 hybrid_clock.cc:648] HybridClock initialized: now 1790393612622071 us; error 48 us; skew 500 ppm
I20260926 03:33:32.622486 12049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:33:32 dist-test-slave-dt7p krb5kdc[11211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:33:32.626122 12049 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:33:32.627786 12049 webserver.cc:460] Webserver started at http://127.0.97.3:36441/ using document root <none> and password file <none>
I20260926 03:33:32.628645 12049 fs_manager.cc:359] Metadata directory not provided
I20260926 03:33:32.628796 12049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:33:32.629918 12049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:33:32 dist-test-slave-dt7p krb5kdc[11211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:33:32.646634 12049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-2/data/instance:
uuid: "a4b3c510809549da93f91a63afd7e636"
format_stamp: "Formatted at 2026-09-26 03:33:32 on dist-test-slave-dt7p"
server_key: "518a45f3f90d9f3922fdd5e39a7aa1d3"
server_key_iv: "21759619398ae898de19e9ed5415a83e"
server_key_version: "kuduclusterkey@0"
I20260926 03:33:32.647621 12049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a4b3c510809549da93f91a63afd7e636"
format_stamp: "Formatted at 2026-09-26 03:33:32 on dist-test-slave-dt7p"
server_key: "518a45f3f90d9f3922fdd5e39a7aa1d3"
server_key_iv: "21759619398ae898de19e9ed5415a83e"
server_key_version: "kuduclusterkey@0"
I20260926 03:33:32.653885 12049 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260926 03:33:32.670812 12064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:33:32.672688 12049 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260926 03:33:32.672904 12049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-2/wal
uuid: "a4b3c510809549da93f91a63afd7e636"
format_stamp: "Formatted at 2026-09-26 03:33:32 on dist-test-slave-dt7p"
server_key: "518a45f3f90d9f3922fdd5e39a7aa1d3"
server_key_iv: "21759619398ae898de19e9ed5415a83e"
server_key_version: "kuduclusterkey@0"
I20260926 03:33:32.673163 12049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:33:32.702766 12049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:33:32.707118 12049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:33:32.707413 12049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:33:32.708487 12049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:33:32.710048 12049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:33:32.710126 12049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:33:32.710215 12049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:33:32.710261 12049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:33:32.758906 12049 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43387
I20260926 03:33:32.758973 12176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43387 every 8 connection(s)
I20260926 03:33:32.760990 12049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:33:32.768543 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 12049
I20260926 03:33:32.768810 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:33:32.769481 388 external_mini_cluster.cc:1428] Setting key 7ba06fd9d327b51308d7ffc9b0508bf9
Sep 26 03:33:32 dist-test-slave-dt7p krb5kdc[11211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:33:32.789703 12179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:33:32.764682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44479 (local address 127.0.97.62:35133)
0926 03:33:32.765375 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0926 03:33:32.765381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:33:32.766610 (+ 1229us) server_negotiation.cc:407] Connection header received
0926 03:33:32.769770 (+ 3160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:33:32.769780 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:33:32.769967 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:33:32.770108 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:33:32.772445 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:33:32.773932 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:33:32.775703 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:33:32.776633 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:33:32.776867 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:33:32.782382 (+ 5515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:33:32.782389 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:33:32.782393 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:33:32.782437 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:33:32.786048 (+ 3611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:33:32.787105 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:33:32.787111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:33:32.787115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:33:32.787255 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:33:32.787699 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:33:32.787706 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:33:32.787710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:33:32.788033 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:33:32.788199 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 03:33:32.789052 (+ 853us) server_negotiation.cc:299] Negotiation successful
0926 03:33:32.789291 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":350,"threads_started":1}
I20260926 03:33:32.790956 12180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:33:32.765466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35133 (local address 127.0.97.3:44479)
0926 03:33:32.766400 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0926 03:33:32.766469 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 03:33:32.769327 (+ 2858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:33:32.770485 (+ 1158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:33:32.770498 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:33:32.771199 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:33:32.772235 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:33:32.772265 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:33:32.774073 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:33:32.774077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:33:32.775572 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:33:32.775585 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:33:32.776910 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:33:32.776916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:33:32.777294 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:33:32.778316 (+ 1022us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:33:32.778344 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:33:32.782018 (+ 3674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:33:32.786217 (+ 4199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:33:32.786245 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:33:32.786271 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:33:32.786988 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:33:32.787374 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:33:32.787379 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:33:32.787383 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:33:32.787590 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:33:32.788184 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:33:32.788193 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:33:32.788674 (+ 481us) client_negotiation.cc:769] Sending connection context
0926 03:33:32.788975 (+ 301us) client_negotiation.cc:240] Negotiation successful
0926 03:33:32.789604 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":373,"threads_started":1}
I20260926 03:33:32.792526 12177 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35133
I20260926 03:33:32.792949 12177 heartbeater.cc:464] Registering TS with master...
I20260926 03:33:32.793798 12177 heartbeater.cc:511] Master 127.0.97.62:35133 requested a full tablet report, sending...
I20260926 03:33:32.795472 11687 ts_manager.cc:194] Registered new tserver with Master: a4b3c510809549da93f91a63afd7e636 (127.0.97.3:43387)
I20260926 03:33:32.796424 11687 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44479
I20260926 03:33:32.799131 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:33:32.817081 12179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:33:32.803914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53702 (local address 127.0.97.62:35133)
0926 03:33:32.804170 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 03:33:32.804180 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:33:32.804637 (+ 457us) server_negotiation.cc:407] Connection header received
0926 03:33:32.804832 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:33:32.804836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:33:32.804931 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:33:32.805014 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:33:32.806624 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:33:32.807423 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:33:32.808854 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:33:32.809329 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:33:32.809430 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:33:32.812006 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:33:32.812011 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:33:32.812014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:33:32.812048 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:33:32.814716 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:33:32.815354 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:33:32.815358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:33:32.815360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:33:32.815485 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:33:32.815786 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:33:32.815789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:33:32.815791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:33:32.816056 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:33:32.816262 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 03:33:32.816527 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 03:33:32.816790 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 03:33:32.822163 11687 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53702:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:33:32.830555 11687 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:33:32.861864 12190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:33:32.845083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33305 (local address 127.0.0.1:35214)
0926 03:33:32.845989 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0926 03:33:32.846047 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 03:33:32.846240 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:33:32.847382 (+ 1142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:33:32.847394 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:33:32.847440 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:33:32.847988 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:33:32.848006 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:33:32.852005 (+ 3999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:33:32.852014 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:33:32.855510 (+ 3496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:33:32.855532 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:33:32.860589 (+ 5057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:33:32.860600 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:33:32.860884 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:33:32.860941 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 03:33:32.861048 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 03:33:32.861262 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":402,"threads_started":1}
I20260926 03:33:32.862208 12193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:33:32.846727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51494 (local address 127.0.97.2:33713)
0926 03:33:32.848664 (+ 1937us) server_negotiation.cc:206] Beginning negotiation
0926 03:33:32.848677 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:33:32.849256 (+ 579us) server_negotiation.cc:407] Connection header received
0926 03:33:32.849372 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:33:32.849379 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:33:32.849628 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:33:32.849842 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:33:32.851042 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:33:32.853167 (+ 2125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:33:32.856776 (+ 3609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:33:32.858926 (+ 2150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:33:32.859096 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:33:32.859408 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0926 03:33:32.861439 (+ 2031us) server_negotiation.cc:299] Negotiation successful
0926 03:33:32.861735 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1739,"thread_start_us":1625,"threads_started":1}
I20260926 03:33:32.862977 12192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:33:32.846446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33713 (local address 127.0.0.1:51494)
0926 03:33:32.848956 (+ 2510us) negotiation.cc:107] Waiting for socket to connect
0926 03:33:32.848971 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:33:32.849119 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:33:32.850158 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:33:32.850167 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:33:32.850202 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:33:32.850764 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:33:32.850793 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:33:32.853457 (+ 2664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:33:32.853465 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:33:32.856519 (+ 3054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:33:32.856541 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:33:32.860170 (+ 3629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:33:32.860181 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:33:32.860540 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:33:32.860645 (+ 105us) client_negotiation.cc:769] Sending connection context
0926 03:33:32.860842 (+ 197us) client_negotiation.cc:240] Negotiation successful
0926 03:33:32.861182 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2415,"mutex_wait_us":1,"thread_start_us":2294,"threads_started":1}
I20260926 03:33:32.865440 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:33:32.845520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35214 (local address 127.0.97.1:33305)
0926 03:33:32.846260 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0926 03:33:32.846271 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:33:32.846318 (+ 47us) server_negotiation.cc:407] Connection header received
0926 03:33:32.846449 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:33:32.846456 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:33:32.846754 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:33:32.846976 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:33:32.848213 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:33:32.851795 (+ 3582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:33:32.857291 (+ 5496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:33:32.860210 (+ 2919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:33:32.864331 (+ 4121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:33:32.864680 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0926 03:33:32.864760 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 03:33:32.865009 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":363,"threads_started":1}
I20260926 03:33:32.868610 11978 tablet_service.cc:1467] Processing CreateTablet for tablet 31eabfa6c7bb4d919609cef57ada0dd5 (DEFAULT_TABLE table=test-table [id=dd9915b77c5f42b5a9833d078ef5704c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:33:32.869449 11835 tablet_service.cc:1467] Processing CreateTablet for tablet 31eabfa6c7bb4d919609cef57ada0dd5 (DEFAULT_TABLE table=test-table [id=dd9915b77c5f42b5a9833d078ef5704c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:33:32.870573 11978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31eabfa6c7bb4d919609cef57ada0dd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:33:32.871521 11835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31eabfa6c7bb4d919609cef57ada0dd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:33:32.871680 12190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:33:32.849105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43387 (local address 127.0.0.1:49322)
0926 03:33:32.862264 (+ 13159us) negotiation.cc:107] Waiting for socket to connect
0926 03:33:32.862293 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 03:33:32.862444 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:33:32.865137 (+ 2693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:33:32.865146 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:33:32.865179 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:33:32.865366 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:33:32.865377 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:33:32.867244 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:33:32.867251 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:33:32.868721 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:33:32.868734 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:33:32.870870 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:33:32.870877 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:33:32.871119 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:33:32.871166 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 03:33:32.871227 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 03:33:32.871359 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13000,"thread_start_us":14335,"threads_started":1}
I20260926 03:33:32.872738 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:33:32.849626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49322 (local address 127.0.97.3:43387)
0926 03:33:32.864278 (+ 14652us) server_negotiation.cc:206] Beginning negotiation
0926 03:33:32.864290 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:33:32.864331 (+ 41us) server_negotiation.cc:407] Connection header received
0926 03:33:32.864455 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:33:32.864463 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:33:32.864872 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:33:32.865166 (+ 294us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:33:32.865575 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:33:32.867091 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:33:32.868885 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:33:32.870588 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:33:32.871977 (+ 1389us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:33:32.872175 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 03:33:32.872234 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 03:33:32.872366 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14438,"thread_start_us":14254,"threads_started":1}
I20260926 03:33:32.876076 12112 tablet_service.cc:1467] Processing CreateTablet for tablet 31eabfa6c7bb4d919609cef57ada0dd5 (DEFAULT_TABLE table=test-table [id=dd9915b77c5f42b5a9833d078ef5704c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:33:32.878121 12112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31eabfa6c7bb4d919609cef57ada0dd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:33:32.888749 12197 tablet_bootstrap.cc:492] T 31eabfa6c7bb4d919609cef57ada0dd5 P d16cd7345a254adc949279844cf52b3b: Bootstrap starting.
I20260926 03:33:32.889880 12196 tablet_bootstrap.cc:492] T 31eabfa6c7bb4d919609cef57ada0dd5 P 88ae8c01e5a245f9a1d413ce104fc8fc: Bootstrap starting.
I20260926 03:33:32.890890 12198 tablet_bootstrap.cc:492] T 31eabfa6c7bb4d919609cef57ada0dd5 P a4b3c510809549da93f91a63afd7e636: Bootstrap starting.
I20260926 03:33:32.893939 12198 tablet_bootstrap.cc:654] T 31eabfa6c7bb4d919609cef57ada0dd5 P a4b3c510809549da93f91a63afd7e636: Neither blocks nor log segments found. Creating new log.
I20260926 03:33:32.894802 12197 tablet_bootstrap.cc:654] T 31eabfa6c7bb4d919609cef57ada0dd5 P d16cd7345a254adc949279844cf52b3b: Neither blocks nor log segments found. Creating new log.
I20260926 03:33:32.895181 12198 log.cc:824] T 31eabfa6c7bb4d919609cef57ada0dd5 P a4b3c510809549da93f91a63afd7e636: Log is configured to *not* fsync() on all Append() calls
I20260926 03:33:32.896097 12197 log.cc:824] T 31eabfa6c7bb4d919609cef57ada0dd5 P d16cd7345a254adc949279844cf52b3b: Log is configured to *not* fsync() on all Append() calls
I20260926 03:33:32.897063 12196 tablet_bootstrap.cc:654] T 31eabfa6c7bb4d919609cef57ada0dd5 P 88ae8c01e5a245f9a1d413ce104fc8fc: Neither blocks nor log segments found. Creating new log.
I20260926 03:33:32.898341 12196 log.cc:824] T 31eabfa6c7bb4d919609cef57ada0dd5 P 88ae8c01e5a245f9a1d413ce104fc8fc: Log is configured to *not* fsync() on all Append() calls
I20260926 03:33:32.899349 12198 tablet_bootstrap.cc:492] T 31eabfa6c7bb4d919609cef57ada0dd5 P a4b3c510809549da93f91a63afd7e636: No bootstrap required, opened a new log
I20260926 03:33:32.899394 12197 tablet_bootstrap.cc:492] T 31eabfa6c7bb4d919609cef57ada0dd5 P d16cd7345a254adc949279844cf52b3b: No bootstrap required, opened a new log
I20260926 03:33:32.899652 12198 ts_tablet_manager.cc:1397] T 31eabfa6c7bb4d919609cef57ada0dd5 P a4b3c510809549da93f91a63afd7e636: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 03:33:32.899672 12197 ts_tablet_manager.cc:1397] T 31eabfa6c7bb4d919609cef57ada0dd5 P d16cd7345a254adc949279844cf52b3b: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.005s
I20260926 03:33:32.901613 12196 tablet_bootstrap.cc:492] T 31eabfa6c7bb4d919609cef57ada0dd5 P 88ae8c01e5a245f9a1d413ce104fc8fc: No bootstrap required, opened a new log
I20260926 03:33:32.901823 12196 ts_tablet_manager.cc:1397] T 31eabfa6c7bb4d919609cef57ada0dd5 P 88ae8c01e5a245f9a1d413ce104fc8fc: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260926 03:33:32.909731 12197 raft_consensus.cc:348] T 31eabfa6c7bb4d919609cef57ada0dd5 P d16cd7345a254adc949279844cf52b3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d16cd7345a254adc949279844cf52b3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33305 } } peers { permanent_uuid: "88ae8c01e5a245f9a1d413ce104fc8fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33713 } } peers { permanent_uuid: "a4b3c510809549da93f91a63afd7e636" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43387 } }
I20260926 03:33:32.910197 12197 raft_consensus.cc:374] T 31eabfa6c7bb4d919609cef57ada0dd5 P d16cd7345a254adc949279844cf52b3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:33:32.910305 12197 raft_consensus.cc:729] T 31eabfa6c7bb4d919609cef57ada0dd5 P d16cd7345a254adc949279844cf52b3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d16cd7345a254adc949279844cf52b3b, State: Initialized, Role: FOLLOWER
I20260926 03:33:32.911121 12197 consensus_queue.cc:260] T 31eabfa6c7bb4d919609cef57ada0dd5 P d16cd7345a254adc949279844cf52b3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d16cd7345a254adc949279844cf52b3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33305 } } peers { permanent_uuid: "88ae8c01e5a245f9a1d413ce104fc8fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33713 } } peers { permanent_uuid: "a4b3c510809549da93f91a63afd7e636" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43387 } }
I20260926 03:33:32.911293 12198 raft_consensus.cc:348] T 31eabfa6c7bb4d919609cef57ada0dd5 P a4b3c510809549da93f91a63afd7e636 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d16cd7345a254adc949279844cf52b3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33305 } } peers { permanent_uuid: "88ae8c01e5a245f9a1d413ce104fc8fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33713 } } peers { permanent_uuid: "a4b3c510809549da93f91a63afd7e636" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43387 } }
I20260926 03:33:32.911744 12198 raft_consensus.cc:374] T 31eabfa6c7bb4d919609cef57ada0dd5 P a4b3c510809549da93f91a63afd7e636 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:33:32.911850 12198 raft_consensus.cc:729] T 31eabfa6c7bb4d919609cef57ada0dd5 P a4b3c510809549da93f91a63afd7e636 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4b3c510809549da93f91a63afd7e636, State: Initialized, Role: FOLLOWER
I20260926 03:33:32.912492 12198 consensus_queue.cc:260] T 31eabfa6c7bb4d919609cef57ada0dd5 P a4b3c510809549da93f91a63afd7e636 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d16cd7345a254adc949279844cf52b3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33305 } } peers { permanent_uuid: "88ae8c01e5a245f9a1d413ce104fc8fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33713 } } peers { permanent_uuid: "a4b3c510809549da93f91a63afd7e636" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43387 } }
I20260926 03:33:32.913024 12196 raft_consensus.cc:348] T 31eabfa6c7bb4d919609cef57ada0dd5 P 88ae8c01e5a245f9a1d413ce104fc8fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d16cd7345a254adc949279844cf52b3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33305 } } peers { permanent_uuid: "88ae8c01e5a245f9a1d413ce104fc8fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33713 } } peers { permanent_uuid: "a4b3c510809549da93f91a63afd7e636" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43387 } }
I20260926 03:33:32.913614 12196 raft_consensus.cc:374] T 31eabfa6c7bb4d919609cef57ada0dd5 P 88ae8c01e5a245f9a1d413ce104fc8fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:33:32.913733 12196 raft_consensus.cc:729] T 31eabfa6c7bb4d919609cef57ada0dd5 P 88ae8c01e5a245f9a1d413ce104fc8fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88ae8c01e5a245f9a1d413ce104fc8fc, State: Initialized, Role: FOLLOWER
I20260926 03:33:32.914642 12197 ts_tablet_manager.cc:1428] T 31eabfa6c7bb4d919609cef57ada0dd5 P d16cd7345a254adc949279844cf52b3b: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260926 03:33:32.914606 12196 consensus_queue.cc:260] T 31eabfa6c7bb4d919609cef57ada0dd5 P 88ae8c01e5a245f9a1d413ce104fc8fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d16cd7345a254adc949279844cf52b3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33305 } } peers { permanent_uuid: "88ae8c01e5a245f9a1d413ce104fc8fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33713 } } peers { permanent_uuid: "a4b3c510809549da93f91a63afd7e636" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43387 } }
I20260926 03:33:32.915908 12177 heartbeater.cc:503] Master 127.0.97.62:35133 was elected leader, sending a full tablet report...
I20260926 03:33:32.916862 12198 ts_tablet_manager.cc:1428] T 31eabfa6c7bb4d919609cef57ada0dd5 P a4b3c510809549da93f91a63afd7e636: Time spent starting tablet: real 0.017s user 0.009s sys 0.010s
I20260926 03:33:32.918305 12196 ts_tablet_manager.cc:1428] T 31eabfa6c7bb4d919609cef57ada0dd5 P 88ae8c01e5a245f9a1d413ce104fc8fc: Time spent starting tablet: real 0.016s user 0.009s sys 0.007s
I20260926 03:33:32.918609 12043 heartbeater.cc:503] Master 127.0.97.62:35133 was elected leader, sending a full tablet report...
W20260926 03:33:33.012554 11901 tablet.cc:2369] T 31eabfa6c7bb4d919609cef57ada0dd5 P d16cd7345a254adc949279844cf52b3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:33:33.012946 12178 tablet.cc:2369] T 31eabfa6c7bb4d919609cef57ada0dd5 P a4b3c510809549da93f91a63afd7e636: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:33:33.068818 12044 tablet.cc:2369] T 31eabfa6c7bb4d919609cef57ada0dd5 P 88ae8c01e5a245f9a1d413ce104fc8fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:33:33.244714 12203 raft_consensus.cc:482] T 31eabfa6c7bb4d919609cef57ada0dd5 P a4b3c510809549da93f91a63afd7e636 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:33:33.245152 12203 raft_consensus.cc:504] T 31eabfa6c7bb4d919609cef57ada0dd5 P a4b3c510809549da93f91a63afd7e636 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d16cd7345a254adc949279844cf52b3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33305 } } peers { permanent_uuid: "88ae8c01e5a245f9a1d413ce104fc8fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33713 } } peers { permanent_uuid: "a4b3c510809549da93f91a63afd7e636" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43387 } }
I20260926 03:33:33.247287 12203 leader_election.cc:288] T 31eabfa6c7bb4d919609cef57ada0dd5 P a4b3c510809549da93f91a63afd7e636 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d16cd7345a254adc949279844cf52b3b (127.0.97.1:33305), 88ae8c01e5a245f9a1d413ce104fc8fc (127.0.97.2:33713)
I20260926 03:33:33.255719 12180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:33:33.247840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33305 (local address 127.0.97.3:58117)
0926 03:33:33.248074 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 03:33:33.248099 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 03:33:33.248302 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:33:33.248994 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:33:33.249001 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:33:33.249035 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:33:33.249328 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:33:33.249340 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:33:33.250959 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:33:33.250965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:33:33.253563 (+ 2598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:33:33.253584 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:33:33.254901 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:33:33.254907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:33:33.255099 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:33:33.255137 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 03:33:33.255233 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 03:33:33.255388 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":22}
I20260926 03:33:33.255730 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:33:33.247822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58117 (local address 127.0.97.1:33305)
0926 03:33:33.248074 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 03:33:33.248081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:33:33.248198 (+ 117us) server_negotiation.cc:407] Connection header received
0926 03:33:33.248542 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:33:33.248548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:33:33.248682 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:33:33.248775 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:33:33.249501 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:33:33.250800 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:33:33.253748 (+ 2948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:33:33.254717 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:33:33.254884 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:33:33.255018 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 03:33:33.255278 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 03:33:33.255444 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 03:33:33.257215 12193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:33:33.248729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38201 (local address 127.0.97.2:33713)
0926 03:33:33.248981 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 03:33:33.248987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:33:33.249595 (+ 608us) server_negotiation.cc:407] Connection header received
0926 03:33:33.249785 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:33:33.249792 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:33:33.249878 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:33:33.249976 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:33:33.250505 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:33:33.251485 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:33:33.254795 (+ 3310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:33:33.255991 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:33:33.256139 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:33:33.256236 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 03:33:33.256791 (+ 555us) server_negotiation.cc:299] Negotiation successful
0926 03:33:33.256939 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 03:33:33.257309 12205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:33:33.248266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33713 (local address 127.0.97.3:38201)
0926 03:33:33.249476 (+ 1210us) negotiation.cc:107] Waiting for socket to connect
0926 03:33:33.249494 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:33:33.249636 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:33:33.249990 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:33:33.249994 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:33:33.250017 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:33:33.250341 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:33:33.250350 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:33:33.251645 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:33:33.251650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:33:33.254609 (+ 2959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:33:33.254623 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:33:33.256382 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:33:33.256388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:33:33.256599 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:33:33.256640 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 03:33:33.256735 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 03:33:33.256883 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1068,"mutex_wait_us":1,"thread_start_us":864,"threads_started":1}
I20260926 03:33:33.257489 11855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31eabfa6c7bb4d919609cef57ada0dd5" candidate_uuid: "a4b3c510809549da93f91a63afd7e636" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d16cd7345a254adc949279844cf52b3b" is_pre_election: true
I20260926 03:33:33.258340 11855 raft_consensus.cc:2454] T 31eabfa6c7bb4d919609cef57ada0dd5 P d16cd7345a254adc949279844cf52b3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4b3c510809549da93f91a63afd7e636 in term 0.
I20260926 03:33:33.258457 11998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31eabfa6c7bb4d919609cef57ada0dd5" candidate_uuid: "a4b3c510809549da93f91a63afd7e636" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88ae8c01e5a245f9a1d413ce104fc8fc" is_pre_election: true
I20260926 03:33:33.259040 11998 raft_consensus.cc:2454] T 31eabfa6c7bb4d919609cef57ada0dd5 P 88ae8c01e5a245f9a1d413ce104fc8fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4b3c510809549da93f91a63afd7e636 in term 0.
I20260926 03:33:33.259248 12066 leader_election.cc:302] T 31eabfa6c7bb4d919609cef57ada0dd5 P a4b3c510809549da93f91a63afd7e636 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4b3c510809549da93f91a63afd7e636, d16cd7345a254adc949279844cf52b3b; no voters:
I20260926 03:33:33.259727 12203 raft_consensus.cc:2781] T 31eabfa6c7bb4d919609cef57ada0dd5 P a4b3c510809549da93f91a63afd7e636 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:33:33.259912 12203 raft_consensus.cc:482] T 31eabfa6c7bb4d919609cef57ada0dd5 P a4b3c510809549da93f91a63afd7e636 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:33:33.260031 12203 raft_consensus.cc:3037] T 31eabfa6c7bb4d919609cef57ada0dd5 P a4b3c510809549da93f91a63afd7e636 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:33:33.262593 12203 raft_consensus.cc:504] T 31eabfa6c7bb4d919609cef57ada0dd5 P a4b3c510809549da93f91a63afd7e636 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d16cd7345a254adc949279844cf52b3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33305 } } peers { permanent_uuid: "88ae8c01e5a245f9a1d413ce104fc8fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33713 } } peers { permanent_uuid: "a4b3c510809549da93f91a63afd7e636" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43387 } }
I20260926 03:33:33.263240 12203 leader_election.cc:288] T 31eabfa6c7bb4d919609cef57ada0dd5 P a4b3c510809549da93f91a63afd7e636 [CANDIDATE]: Term 1 election: Requested vote from peers d16cd7345a254adc949279844cf52b3b (127.0.97.1:33305), 88ae8c01e5a245f9a1d413ce104fc8fc (127.0.97.2:33713)
I20260926 03:33:33.263832 11855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31eabfa6c7bb4d919609cef57ada0dd5" candidate_uuid: "a4b3c510809549da93f91a63afd7e636" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d16cd7345a254adc949279844cf52b3b"
I20260926 03:33:33.264060 11855 raft_consensus.cc:3037] T 31eabfa6c7bb4d919609cef57ada0dd5 P d16cd7345a254adc949279844cf52b3b [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:33:33.264173 11998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31eabfa6c7bb4d919609cef57ada0dd5" candidate_uuid: "a4b3c510809549da93f91a63afd7e636" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88ae8c01e5a245f9a1d413ce104fc8fc"
I20260926 03:33:33.264427 11998 raft_consensus.cc:3037] T 31eabfa6c7bb4d919609cef57ada0dd5 P 88ae8c01e5a245f9a1d413ce104fc8fc [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:33:33.266726 11855 raft_consensus.cc:2454] T 31eabfa6c7bb4d919609cef57ada0dd5 P d16cd7345a254adc949279844cf52b3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4b3c510809549da93f91a63afd7e636 in term 1.
I20260926 03:33:33.267268 12066 leader_election.cc:302] T 31eabfa6c7bb4d919609cef57ada0dd5 P a4b3c510809549da93f91a63afd7e636 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4b3c510809549da93f91a63afd7e636, d16cd7345a254adc949279844cf52b3b; no voters:
I20260926 03:33:33.267621 12203 raft_consensus.cc:2781] T 31eabfa6c7bb4d919609cef57ada0dd5 P a4b3c510809549da93f91a63afd7e636 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:33:33.267910 11998 raft_consensus.cc:2454] T 31eabfa6c7bb4d919609cef57ada0dd5 P 88ae8c01e5a245f9a1d413ce104fc8fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4b3c510809549da93f91a63afd7e636 in term 1.
I20260926 03:33:33.268183 12203 raft_consensus.cc:686] T 31eabfa6c7bb4d919609cef57ada0dd5 P a4b3c510809549da93f91a63afd7e636 [term 1 LEADER]: Becoming Leader. State: Replica: a4b3c510809549da93f91a63afd7e636, State: Running, Role: LEADER
I20260926 03:33:33.268936 12203 consensus_queue.cc:237] T 31eabfa6c7bb4d919609cef57ada0dd5 P a4b3c510809549da93f91a63afd7e636 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d16cd7345a254adc949279844cf52b3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33305 } } peers { permanent_uuid: "88ae8c01e5a245f9a1d413ce104fc8fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33713 } } peers { permanent_uuid: "a4b3c510809549da93f91a63afd7e636" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43387 } }
I20260926 03:33:33.275352 11685 catalog_manager.cc:5495] T 31eabfa6c7bb4d919609cef57ada0dd5 P a4b3c510809549da93f91a63afd7e636 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4b3c510809549da93f91a63afd7e636 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "a4b3c510809549da93f91a63afd7e636" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d16cd7345a254adc949279844cf52b3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88ae8c01e5a245f9a1d413ce104fc8fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4b3c510809549da93f91a63afd7e636" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43387 } health_report { overall_health: HEALTHY } } }
I20260926 03:33:33.321329 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:33:33.315373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49330 (local address 127.0.97.3:43387)
0926 03:33:33.315586 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 03:33:33.315591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:33:33.315611 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:33:33.315803 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:33:33.315807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:33:33.315910 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:33:33.316051 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:33:33.316421 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:33:33.317448 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:33:33.318950 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:33:33.319366 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:33:33.319473 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:33:33.320018 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:33:33.320811 (+ 793us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:33:33.320934 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 03:33:33.321015 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 03:33:33.321092 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 03:33:33.331573 11855 raft_consensus.cc:1261] T 31eabfa6c7bb4d919609cef57ada0dd5 P d16cd7345a254adc949279844cf52b3b [term 1 FOLLOWER]: Refusing update from remote peer a4b3c510809549da93f91a63afd7e636: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:33:33.331573 11998 raft_consensus.cc:1261] T 31eabfa6c7bb4d919609cef57ada0dd5 P 88ae8c01e5a245f9a1d413ce104fc8fc [term 1 FOLLOWER]: Refusing update from remote peer a4b3c510809549da93f91a63afd7e636: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:33:33.332687 12206 consensus_queue.cc:1035] T 31eabfa6c7bb4d919609cef57ada0dd5 P a4b3c510809549da93f91a63afd7e636 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88ae8c01e5a245f9a1d413ce104fc8fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:33:33.333105 12203 consensus_queue.cc:1035] T 31eabfa6c7bb4d919609cef57ada0dd5 P a4b3c510809549da93f91a63afd7e636 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d16cd7345a254adc949279844cf52b3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:33:33.348001 12211 mvcc.cc:205] Tried to move back new op lower bound from 7333452240194191360 to 7333452239954718720. Current Snapshot: MvccSnapshot[applied={T|T < 7333452240194191360}]
I20260926 03:33:33.362528 11687 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:33:33.362887 11687 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:33:33.366910 11687 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6591e6134b054f86900ecf36692f079e: Sending DeleteTablet for 3 replicas of tablet 31eabfa6c7bb4d919609cef57ada0dd5
I20260926 03:33:33.368270 11835 tablet_service.cc:1514] Processing DeleteTablet for tablet 31eabfa6c7bb4d919609cef57ada0dd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:33:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35214
I20260926 03:33:33.368654 12112 tablet_service.cc:1514] Processing DeleteTablet for tablet 31eabfa6c7bb4d919609cef57ada0dd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:33:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49322
I20260926 03:33:33.368752 11978 tablet_service.cc:1514] Processing DeleteTablet for tablet 31eabfa6c7bb4d919609cef57ada0dd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:33:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51494
I20260926 03:33:33.374957 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 11767
I20260926 03:33:33.378383 12219 tablet_replica.cc:330] T 31eabfa6c7bb4d919609cef57ada0dd5 P 88ae8c01e5a245f9a1d413ce104fc8fc: stopping tablet replica
I20260926 03:33:33.379142 12219 raft_consensus.cc:2229] T 31eabfa6c7bb4d919609cef57ada0dd5 P 88ae8c01e5a245f9a1d413ce104fc8fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:33:33.379943 12219 raft_consensus.cc:2258] T 31eabfa6c7bb4d919609cef57ada0dd5 P 88ae8c01e5a245f9a1d413ce104fc8fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:33:33.385669 12219 ts_tablet_manager.cc:1905] T 31eabfa6c7bb4d919609cef57ada0dd5 P 88ae8c01e5a245f9a1d413ce104fc8fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:33:33.387060 12220 tablet_replica.cc:330] T 31eabfa6c7bb4d919609cef57ada0dd5 P a4b3c510809549da93f91a63afd7e636: stopping tablet replica
I20260926 03:33:33.387627 12220 raft_consensus.cc:2229] T 31eabfa6c7bb4d919609cef57ada0dd5 P a4b3c510809549da93f91a63afd7e636 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:33:33.388480 12220 raft_consensus.cc:2258] T 31eabfa6c7bb4d919609cef57ada0dd5 P a4b3c510809549da93f91a63afd7e636 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:33:33.394132 12220 ts_tablet_manager.cc:1905] T 31eabfa6c7bb4d919609cef57ada0dd5 P a4b3c510809549da93f91a63afd7e636: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:33:33.394641 12219 ts_tablet_manager.cc:1918] T 31eabfa6c7bb4d919609cef57ada0dd5 P 88ae8c01e5a245f9a1d413ce104fc8fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:33:33.394891 12219 log.cc:1196] T 31eabfa6c7bb4d919609cef57ada0dd5 P 88ae8c01e5a245f9a1d413ce104fc8fc: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-1/wal/wals/31eabfa6c7bb4d919609cef57ada0dd5
I20260926 03:33:33.395541 12219 ts_tablet_manager.cc:1939] T 31eabfa6c7bb4d919609cef57ada0dd5 P 88ae8c01e5a245f9a1d413ce104fc8fc: Deleting consensus metadata
I20260926 03:33:33.398070 11672 catalog_manager.cc:4841] TS 88ae8c01e5a245f9a1d413ce104fc8fc (127.0.97.2:33713): tablet 31eabfa6c7bb4d919609cef57ada0dd5 (table test-table [id=dd9915b77c5f42b5a9833d078ef5704c]) successfully deleted
I20260926 03:33:33.403915 12220 ts_tablet_manager.cc:1918] T 31eabfa6c7bb4d919609cef57ada0dd5 P a4b3c510809549da93f91a63afd7e636: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:33:33.404215 12220 log.cc:1196] T 31eabfa6c7bb4d919609cef57ada0dd5 P a4b3c510809549da93f91a63afd7e636: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-2/wal/wals/31eabfa6c7bb4d919609cef57ada0dd5
I20260926 03:33:33.406304 12220 ts_tablet_manager.cc:1939] T 31eabfa6c7bb4d919609cef57ada0dd5 P a4b3c510809549da93f91a63afd7e636: Deleting consensus metadata
I20260926 03:33:33.408735 11672 catalog_manager.cc:4841] TS a4b3c510809549da93f91a63afd7e636 (127.0.97.3:43387): tablet 31eabfa6c7bb4d919609cef57ada0dd5 (table test-table [id=dd9915b77c5f42b5a9833d078ef5704c]) successfully deleted
I20260926 03:33:33.409727 12218 tablet_replica.cc:330] T 31eabfa6c7bb4d919609cef57ada0dd5 P d16cd7345a254adc949279844cf52b3b: stopping tablet replica
I20260926 03:33:33.410385 12218 raft_consensus.cc:2229] T 31eabfa6c7bb4d919609cef57ada0dd5 P d16cd7345a254adc949279844cf52b3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:33:33.411114 12218 raft_consensus.cc:2258] T 31eabfa6c7bb4d919609cef57ada0dd5 P d16cd7345a254adc949279844cf52b3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:33:33.414093 12218 ts_tablet_manager.cc:1905] T 31eabfa6c7bb4d919609cef57ada0dd5 P d16cd7345a254adc949279844cf52b3b: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 03:33:33 dist-test-slave-dt7p krb5kdc[11211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:33:33.423869 12218 ts_tablet_manager.cc:1918] T 31eabfa6c7bb4d919609cef57ada0dd5 P d16cd7345a254adc949279844cf52b3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:33:33.424059 12218 log.cc:1196] T 31eabfa6c7bb4d919609cef57ada0dd5 P d16cd7345a254adc949279844cf52b3b: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790393463219689-388-0/minicluster-data/ts-0/wal/wals/31eabfa6c7bb4d919609cef57ada0dd5
I20260926 03:33:33.424638 12218 ts_tablet_manager.cc:1939] T 31eabfa6c7bb4d919609cef57ada0dd5 P d16cd7345a254adc949279844cf52b3b: Deleting consensus metadata
I20260926 03:33:33.426676 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:33:33.376689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35220 (local address 127.0.97.1:33305)
0926 03:33:33.376970 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0926 03:33:33.376981 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:33:33.386383 (+ 9402us) server_negotiation.cc:407] Connection header received
0926 03:33:33.387018 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:33:33.387027 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:33:33.387148 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:33:33.387251 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:33:33.406564 (+ 19313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:33:33.408214 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:33:33.410661 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:33:33.411395 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:33:33.411498 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:33:33.417680 (+ 6182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:33:33.417699 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:33:33.417725 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:33:33.417776 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:33:33.422595 (+ 4819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:33:33.423618 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:33:33.423630 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:33:33.423634 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:33:33.423806 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:33:33.424498 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:33:33.424503 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:33:33.424506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:33:33.425127 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:33:33.425474 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0926 03:33:33.425837 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 03:33:33.426087 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 03:33:33.427008 11672 catalog_manager.cc:4841] TS d16cd7345a254adc949279844cf52b3b (127.0.97.1:33305): tablet 31eabfa6c7bb4d919609cef57ada0dd5 (table test-table [id=dd9915b77c5f42b5a9833d078ef5704c]) successfully deleted
I20260926 03:33:33.427743 11895 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:33:33.527238 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 11767
I20260926 03:33:33.548800 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 11906
Sep 26 03:33:33 dist-test-slave-dt7p krb5kdc[11211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:33:33.569602 12193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:33:33.549771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51508 (local address 127.0.97.2:33713)
0926 03:33:33.549969 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 03:33:33.549974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:33:33.549995 (+ 21us) server_negotiation.cc:407] Connection header received
0926 03:33:33.550232 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:33:33.550237 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:33:33.550313 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:33:33.550409 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:33:33.552354 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:33:33.553477 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:33:33.555494 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:33:33.555970 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:33:33.556081 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:33:33.561867 (+ 5786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:33:33.561895 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:33:33.561926 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:33:33.562004 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:33:33.566274 (+ 4270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:33:33.567097 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:33:33.567109 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:33:33.567113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:33:33.567292 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:33:33.567660 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:33:33.567665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:33:33.567668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:33:33.568238 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:33:33.568623 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0926 03:33:33.568901 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 03:33:33.569158 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 03:33:33.570487 12038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:33:33.670105 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 11906
I20260926 03:33:33.690440 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 12049
Sep 26 03:33:33 dist-test-slave-dt7p krb5kdc[11211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:33:33.710965 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:33:33.691369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49344 (local address 127.0.97.3:43387)
0926 03:33:33.691573 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 03:33:33.691577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:33:33.691636 (+ 59us) server_negotiation.cc:407] Connection header received
0926 03:33:33.691887 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:33:33.691892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:33:33.691967 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:33:33.692070 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:33:33.694090 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:33:33.694959 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:33:33.697095 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:33:33.697582 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:33:33.697699 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:33:33.703495 (+ 5796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:33:33.703512 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:33:33.703533 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:33:33.703580 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:33:33.707325 (+ 3745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:33:33.708229 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:33:33.708239 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:33:33.708242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:33:33.708416 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:33:33.709005 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:33:33.709016 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:33:33.709018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:33:33.709489 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:33:33.709896 (+ 407us) server_negotiation.cc:1035] Waiting for connection context
0926 03:33:33.710244 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 03:33:33.710520 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 03:33:33.711817 12172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:33:33.857121 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 12049
I20260926 03:33:33.879444 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 11655
I20260926 03:33:33.880926 11717 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:33:34.014564 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 11655
I20260926 03:33:34.039685 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 03:33:34.386246 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 03:33:34.386516 388 mini_ranger.cc:66] Stopping Ranger...
I20260926 03:33:34.743775 388 mini_ranger.cc:68] Stopped Ranger
2026-09-26 03:33:34.744 UTC [11241] LOG: received smart shutdown request
2026-09-26 03:33:34.748 UTC [11241] LOG: background worker "logical replication launcher" (PID 11250) exited with exit code 1
2026-09-26 03:33:34.748 UTC [11245] LOG: shutting down
2026-09-26 03:33:34.968 UTC [11241] LOG: database system is shut down
2026-09-26T03:33:34Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (53847 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:33:35 dist-test-slave-dt7p krb5kdc[12234](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:33:35 dist-test-slave-dt7p krb5kdc[12234](info): set up 2 sockets
Sep 26 03:33:35 dist-test-slave-dt7p krb5kdc[12234](info): commencing operation
krb5kdc: starting...
W20260926 03:33:36.273514 388 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:33:36 dist-test-slave-dt7p krb5kdc[12234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:33:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:33:36Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260926 03:33:36.689393 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskssCcLE/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/postgres -l logfile start
2026-09-26 03:33:39.507 UTC [12264] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 03:33:39.507 UTC [12264] LOG: listening on IPv4 address "127.0.97.20", port 44695
2026-09-26 03:33:39.512 UTC [12264] LOG: listening on Unix socket "/tmp/.s.PGSQL.44695"
2026-09-26 03:33:39.537 UTC [12267] LOG: database system was shut down at 2026-09-26 03:33:37 UTC
2026-09-26 03:33:39.541 UTC [12264] LOG: database system is ready to accept connections
I20260926 03:33:40.533140 388 mini_postgres.cc:96] Postgres bound to 44695
2026-09-26 03:33:40.554 UTC [12275] FATAL: database "slave" does not exist
127.0.97.20:44695 - accepting connections
I20260926 03:33:40.554778 388 mini_ranger.cc:161] Starting Ranger...
I20260926 03:33:40.585866 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 03:33:41.143446 388 mini_ranger.cc:87] Created ranger Postgres database
I20260926 03:33:41.143594 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ranger-admin
2026-09-26 03:33:41,478 [I] DB FLAVOR :POSTGRES
2026-09-26 03:33:41,479 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 03:33:41,479 [I] Checking connection..
2026-09-26 03:33:41,479 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 03:33:41,799 [I] Checking connection passed.
2026-09-26 03:33:41,799 [I] --------- Verifying version history table ---------
2026-09-26 03:33:41,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 03:33:42,112 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 03:33:42,113 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 03:33:42,436 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 03:33:42,437 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 03:33:42,437 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 03:33:42.756 UTC [12339] WARNING: there is no transaction in progress
2026-09-26 03:33:42,765 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 03:33:42,766 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 03:33:43,125 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 03:33:43,126 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 03:33:43,126 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 03:33:43,437 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 03:33:43,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-dt7p.c.gcp-upstream.internal','N') ;"
2026-09-26 03:33:43,948 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 03:33:44,234 [I] Table x_portal_user does not exist in database ranger
2026-09-26 03:33:44,235 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 03:33:44,525 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 03:33:44,525 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 03:33:44,787 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 03:33:45,077 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 03:33:45,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 03:33:46.195 UTC [12447] WARNING: there is no transaction in progress
2026-09-26 03:33:46.214 UTC [12447] WARNING: there is no transaction in progress
2026-09-26 03:33:46.232 UTC [12447] WARNING: there is no transaction in progress
2026-09-26 03:33:46.250 UTC [12447] WARNING: there is no transaction in progress
2026-09-26 03:33:46.268 UTC [12447] WARNING: there is no transaction in progress
2026-09-26 03:33:46.292 UTC [12447] WARNING: there is no transaction in progress
2026-09-26 03:33:46.355 UTC [12447] WARNING: there is no transaction in progress
2026-09-26 03:33:46.363 UTC [12447] WARNING: there is no transaction in progress
2026-09-26 03:33:46.380 UTC [12447] WARNING: there is no transaction in progress
2026-09-26 03:33:46.393 UTC [12447] WARNING: there is no transaction in progress
2026-09-26 03:33:46.405 UTC [12447] WARNING: there is no transaction in progress
2026-09-26 03:33:46.423 UTC [12447] WARNING: there is no transaction in progress
2026-09-26 03:33:46.434 UTC [12447] WARNING: there is no transaction in progress
2026-09-26 03:33:46.444 UTC [12447] WARNING: there is no transaction in progress
2026-09-26 03:33:46.454 UTC [12447] WARNING: there is no transaction in progress
2026-09-26 03:33:46.463 UTC [12447] WARNING: there is no transaction in progress
2026-09-26 03:33:47,235 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 03:33:47,236 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 03:33:47,504 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 03:33:47,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 03:33:47,814 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 03:33:47,815 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 03:33:48,092 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 03:33:48,092 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 03:33:48,376 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 03:33:48,663 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-dt7p.c.gcp-upstream.internal';"
2026-09-26 03:33:48,937 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 03:33:48,937 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 03:33:49,209 [I] DB_PATCHES have already been applied
I20260926 03:33:49.217306 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskssCcLE/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 03:33:49.217453 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260926 03:33:49.236266 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:54829
Sep 26, 2026 3:33:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 3:33:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 3:33:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 3:33:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 3:33:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 3:33:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 3:33:50 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 03:33:50.443464 388 mini_ranger.cc:160] Time spent starting Ranger: real 9.889s user 0.007s sys 0.046s
Sep 26, 2026 3:34:12 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 3:34:12 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 3:34:12 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 3:34:12 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 3:34:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 3:34:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 3:34:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 3:34:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 3:34:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 3:34:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 3:34:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 3:34:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 3:34:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 3:34:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 3:34:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 3:34:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 3:34:12 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 3:34:13 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 03:34:14.232300 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 03:34:14 dist-test-slave-dt7p krb5kdc[12234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393654, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 03:34:14.425272 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 03:34:14.451208 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 03:34:15.407842 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 03:34:15.407987 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ranger-kms
I20260926 03:34:15.408034 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:44695
2026-09-26 03:34:15,838 [I] DB FLAVOR :POSTGRES
2026-09-26 03:34:15,838 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 03:34:15,838 [I] Checking connection
2026-09-26 03:34:15,838 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44695/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 03:34:16,119 [I] connection success
2026-09-26 03:34:16,119 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 03:34:16,119 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 03:34:16,119 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44695/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 03:34:16,375 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 03:34:16,375 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 03:34:16,376 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 03:34:16,376 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:44695/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 03:34:16,669 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 03:34:17.166849 388 mini_ranger_kms.cc:310] Created kms service
I20260926 03:34:17.351336 388 mini_ranger_kms.cc:326] Created kudu user
I20260926 03:34:17.407363 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 03:34:17 dist-test-slave-dt7p krb5kdc[12234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393654, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:34:17.873134 388 mini_ranger_kms.cc:384] Added ranger policy
I20260926 03:34:17.873553 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskssCcLE/build/asan/bin/postgresql.jar:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskssCcLE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 03:34:17.873631 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260926 03:34:17.893380 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12628
I20260926 03:34:17.893523 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:60423
Sep 26, 2026 3:34:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 3:34:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 3:34:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 3:34:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 3:34:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 3:34:19 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 03:34:20.330304 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.905s user 0.069s sys 0.386s
I20260926 03:34:20.330614 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 03:34:20.330811 388 mini_ranger_kms.cc:401] 127.0.97.20:60423/kms/v1/keys
Sep 26 03:34:20 dist-test-slave-dt7p krb5kdc[12234](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790393660, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 03:34:20 dist-test-slave-dt7p krb5kdc[12234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393660, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 26 03:34:21 dist-test-slave-dt7p krb5kdc[12234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393660, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 26 03:34:23 dist-test-slave-dt7p krb5kdc[12234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:34:23.763237 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37003
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37003
--ranger_config_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:60423/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:34:24.152122 12678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:34:24.152547 12678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:34:24.152762 12678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:34:24.160342 12678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:34:24.160496 12678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:34:24.160629 12678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:34:24.160742 12678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:34:24.160841 12678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:34:24.171556 12678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:60423/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37003
--ranger_config_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:34:24.173131 12678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:34:24.174683 12678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:34:24.188133 12683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:24.188076 12684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:24.188103 12686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:34:24.188529 12678 server_base.cc:1034] running on GCE node
I20260926 03:34:24.189586 12678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:34:24.191102 12678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:34:24.192303 12678 hybrid_clock.cc:648] HybridClock initialized: now 1790393664192260 us; error 32 us; skew 500 ppm
I20260926 03:34:24.192695 12678 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:34:24 dist-test-slave-dt7p krb5kdc[12234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:34:24.196161 12678 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:34:24.197750 12678 webserver.cc:460] Webserver started at http://127.0.97.62:33765/ using document root <none> and password file <none>
I20260926 03:34:24.198541 12678 fs_manager.cc:359] Metadata directory not provided
I20260926 03:34:24.198673 12678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:34:24.199031 12678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:34:24 dist-test-slave-dt7p krb5kdc[12234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:34:24.266737 12678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "7f2eb46af1d04155923f8023f100f9a0"
format_stamp: "Formatted at 2026-09-26 03:34:24 on dist-test-slave-dt7p"
server_key: "33f3b913d42b9a9164bb638495f24303"
server_key_iv: "9a39c75e385bc4f24775ea6236f793f7"
server_key_version: "kuduclusterkey@0"
I20260926 03:34:24.267720 12678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "7f2eb46af1d04155923f8023f100f9a0"
format_stamp: "Formatted at 2026-09-26 03:34:24 on dist-test-slave-dt7p"
server_key: "33f3b913d42b9a9164bb638495f24303"
server_key_iv: "9a39c75e385bc4f24775ea6236f793f7"
server_key_version: "kuduclusterkey@0"
I20260926 03:34:24.274008 12678 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260926 03:34:24.297257 12693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:24.299329 12678 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 03:34:24.299597 12678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "7f2eb46af1d04155923f8023f100f9a0"
format_stamp: "Formatted at 2026-09-26 03:34:24 on dist-test-slave-dt7p"
server_key: "33f3b913d42b9a9164bb638495f24303"
server_key_iv: "9a39c75e385bc4f24775ea6236f793f7"
server_key_version: "kuduclusterkey@0"
I20260926 03:34:24.299913 12678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:34:24.325336 12678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:34:24.330775 12678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:34:24.331147 12678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:34:24.354439 12678 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37003
I20260926 03:34:24.354439 12744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37003 every 8 connection(s)
I20260926 03:34:24.356487 12678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:34:24.361166 12745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:34:24.365446 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 12678
I20260926 03:34:24.365710 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:34:24.366592 388 external_mini_cluster.cc:1428] Setting key 19d99339fe01b0bb4e9149aebfd86929
I20260926 03:34:24.378358 12745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0: Bootstrap starting.
I20260926 03:34:24.384348 12745 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0: Neither blocks nor log segments found. Creating new log.
I20260926 03:34:24.385859 12745 log.cc:824] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0: Log is configured to *not* fsync() on all Append() calls
I20260926 03:34:24.389952 12745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0: No bootstrap required, opened a new log
Sep 26 03:34:24 dist-test-slave-dt7p krb5kdc[12234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:34:24.399906 12745 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f2eb46af1d04155923f8023f100f9a0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37003 } }
I20260926 03:34:24.400511 12745 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:34:24.400689 12745 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f2eb46af1d04155923f8023f100f9a0, State: Initialized, Role: FOLLOWER
I20260926 03:34:24.401535 12745 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f2eb46af1d04155923f8023f100f9a0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37003 } }
I20260926 03:34:24.401835 12745 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:34:24.402016 12745 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:34:24.402231 12745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:34:24.405380 12745 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f2eb46af1d04155923f8023f100f9a0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37003 } }
I20260926 03:34:24.406121 12745 leader_election.cc:302] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f2eb46af1d04155923f8023f100f9a0; no voters:
I20260926 03:34:24.406986 12745 leader_election.cc:288] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:34:24.407089 12750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:34:24.409693 12745 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:34:24.410925 12748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:24.369114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57652 (local address 127.0.97.62:37003)
0926 03:34:24.383890 (+ 14776us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:24.383908 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:24.384257 (+ 349us) server_negotiation.cc:407] Connection header received
0926 03:34:24.387075 (+ 2818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:24.387112 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:24.387724 (+ 612us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:24.388241 (+ 517us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:24.390178 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:24.392368 (+ 2190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:24.394324 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:24.395488 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:24.395694 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:24.401089 (+ 5395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:24.401100 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:24.401126 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:24.401187 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:24.405256 (+ 4069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:24.406300 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:24.406309 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:24.406319 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:24.406530 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:24.407031 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:24.407037 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:24.407040 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:24.407924 (+ 884us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:24.408233 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:24.408750 (+ 517us) server_negotiation.cc:299] Negotiation successful
0926 03:34:24.409190 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14439,"thread_start_us":14688,"threads_started":1}
I20260926 03:34:24.416777 12750 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0 [term 1 LEADER]: Becoming Leader. State: Replica: 7f2eb46af1d04155923f8023f100f9a0, State: Running, Role: LEADER
I20260926 03:34:24.417387 12750 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f2eb46af1d04155923f8023f100f9a0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37003 } }
I20260926 03:34:24.424299 12752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f2eb46af1d04155923f8023f100f9a0. Latest consensus state: current_term: 1 leader_uuid: "7f2eb46af1d04155923f8023f100f9a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f2eb46af1d04155923f8023f100f9a0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37003 } } }
I20260926 03:34:24.424721 12752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:34:24.425336 12751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f2eb46af1d04155923f8023f100f9a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f2eb46af1d04155923f8023f100f9a0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37003 } } }
I20260926 03:34:24.425606 12751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:34:24.428661 12756 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:34:24.436210 12756 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:34:24.455636 12756 catalog_manager.cc:1347] Generated new cluster ID: 2746431caede439b81ca2dab88174f8d
I20260926 03:34:24.455790 12756 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:34:24.473989 12756 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:34:24.476027 12756 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:34:24.487306 12756 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0: Generated new TSK 0
I20260926 03:34:24.488536 12756 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:34:24.492816 12745 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:34:25.389567 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:60423/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37003
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:34:25 dist-test-slave-dt7p krb5kdc[12234](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790393665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 03:34:25.897135 12790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:34:25.897485 12790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:34:25.897647 12790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:34:25.909752 12790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:34:25.909994 12790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:34:25.910210 12790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:34:25.922026 12790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:60423/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:34:25.923786 12790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:34:25.925719 12790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:34:25.947517 12800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:25.947532 12801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:34:25.955525 12790 server_base.cc:1034] running on GCE node
W20260926 03:34:25.956017 12803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:34:25.957119 12790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:34:25.959820 12790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:34:25.961081 12790 hybrid_clock.cc:648] HybridClock initialized: now 1790393665961004 us; error 63 us; skew 500 ppm
I20260926 03:34:25.961524 12790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:34:25 dist-test-slave-dt7p krb5kdc[12234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:34:25.966543 12790 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:34:25.970957 12790 webserver.cc:460] Webserver started at http://127.0.97.1:40201/ using document root <none> and password file <none>
I20260926 03:34:25.972137 12790 fs_manager.cc:359] Metadata directory not provided
I20260926 03:34:25.972294 12790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:34:25.972745 12790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:34:25 dist-test-slave-dt7p krb5kdc[12234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:34:26.001405 12790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-0/data/instance:
uuid: "e2a557024c9b420a975c3abf0f64a40c"
format_stamp: "Formatted at 2026-09-26 03:34:25 on dist-test-slave-dt7p"
server_key: "5a4eb17e59153422b08c4cc1abd9673b"
server_key_iv: "e0146bc238645bcd042575b9c3a92e3f"
server_key_version: "kuduclusterkey@0"
I20260926 03:34:26.002728 12790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e2a557024c9b420a975c3abf0f64a40c"
format_stamp: "Formatted at 2026-09-26 03:34:25 on dist-test-slave-dt7p"
server_key: "5a4eb17e59153422b08c4cc1abd9673b"
server_key_iv: "e0146bc238645bcd042575b9c3a92e3f"
server_key_version: "kuduclusterkey@0"
I20260926 03:34:26.011904 12790 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s
I20260926 03:34:26.055768 12810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:26.058111 12790 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.006s sys 0.005s
I20260926 03:34:26.058414 12790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-0/wal
uuid: "e2a557024c9b420a975c3abf0f64a40c"
format_stamp: "Formatted at 2026-09-26 03:34:25 on dist-test-slave-dt7p"
server_key: "5a4eb17e59153422b08c4cc1abd9673b"
server_key_iv: "e0146bc238645bcd042575b9c3a92e3f"
server_key_version: "kuduclusterkey@0"
I20260926 03:34:26.058801 12790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:34:26.094408 12790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:34:26.102065 12790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:34:26.102592 12790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:34:26.106796 12790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:34:26.109194 12790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:34:26.109308 12790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:26.109452 12790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:34:26.109527 12790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:34:26 dist-test-slave-dt7p krb5kdc[12234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:34:26.234000 12790 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40605
I20260926 03:34:26.234026 12922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40605 every 8 connection(s)
I20260926 03:34:26.235658 12790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:34:26.241580 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 12790
I20260926 03:34:26.241875 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:34:26.242919 388 external_mini_cluster.cc:1428] Setting key 70649b54733f1e089aa666eb81f34d11
Sep 26 03:34:26 dist-test-slave-dt7p krb5kdc[12234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:34:26.284350 12925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:26.241060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55005 (local address 127.0.97.62:37003)
0926 03:34:26.264218 (+ 23158us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:26.264229 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:26.264265 (+ 36us) server_negotiation.cc:407] Connection header received
0926 03:34:26.267857 (+ 3592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:26.267864 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:26.268002 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:26.268155 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:26.270265 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:26.271147 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:26.273137 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:26.273596 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:26.273742 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:26.277656 (+ 3914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:26.277662 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:26.277665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:26.277710 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:26.280890 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:26.281753 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:26.281758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:26.281761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:26.281891 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:26.282357 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:26.282367 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:26.282369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:26.282675 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:26.282851 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:26.283646 (+ 795us) server_negotiation.cc:299] Negotiation successful
0926 03:34:26.283916 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22927,"thread_start_us":22760,"threads_started":1}
I20260926 03:34:26.285728 12926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:26.241414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37003 (local address 127.0.97.1:55005)
0926 03:34:26.261720 (+ 20306us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:26.261813 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0926 03:34:26.266592 (+ 4779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:34:26.268485 (+ 1893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:34:26.268496 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:34:26.269137 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:34:26.269876 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:26.269896 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:26.271360 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:26.271372 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:26.272975 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:26.272989 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:26.273858 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:26.273867 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:26.274122 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:26.274938 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:34:26.274963 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:34:26.277391 (+ 2428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:34:26.281035 (+ 3644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:26.281046 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:26.281069 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:26.281614 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:26.282004 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:26.282010 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:26.282014 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:26.282234 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:26.282822 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:34:26.282831 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:34:26.283247 (+ 416us) client_negotiation.cc:769] Sending connection context
0926 03:34:26.283554 (+ 307us) client_negotiation.cc:240] Negotiation successful
0926 03:34:26.284180 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19770,"thread_start_us":19415,"threads_started":1}
I20260926 03:34:26.287801 12923 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37003
I20260926 03:34:26.288183 12923 heartbeater.cc:464] Registering TS with master...
I20260926 03:34:26.289103 12923 heartbeater.cc:511] Master 127.0.97.62:37003 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 03:34:26.291597 12710 ts_manager.cc:194] Registered new tserver with Master: e2a557024c9b420a975c3abf0f64a40c (127.0.97.1:40605)
I20260926 03:34:26.293591 12710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55005
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:34:26.328693 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:60423/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37003
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:34:26 dist-test-slave-dt7p krb5kdc[12234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260926 03:34:26.906949 12929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:34:26.907629 12929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:34:26.907893 12929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:34:26.925190 12929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:34:26.925508 12929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:34:26.925863 12929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:34:26.949183 12929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:60423/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:34:26.952158 12929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:34:26.954640 12929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:34:26.983949 12937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:26.984441 12938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:26.987870 12941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:34:26.988161 12929 server_base.cc:1034] running on GCE node
I20260926 03:34:26.989274 12929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:34:26.990808 12929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:34:26.992122 12929 hybrid_clock.cc:648] HybridClock initialized: now 1790393666992035 us; error 78 us; skew 500 ppm
I20260926 03:34:26.992630 12929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:34:26 dist-test-slave-dt7p krb5kdc[12234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:34:26.997555 12929 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:34:26.999905 12929 webserver.cc:460] Webserver started at http://127.0.97.2:36379/ using document root <none> and password file <none>
I20260926 03:34:27.000931 12929 fs_manager.cc:359] Metadata directory not provided
I20260926 03:34:27.001077 12929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:34:27.001470 12929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:34:27 dist-test-slave-dt7p krb5kdc[12234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:34:27.023747 12929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-1/data/instance:
uuid: "bdef2b583f0640108633f2970851619c"
format_stamp: "Formatted at 2026-09-26 03:34:27 on dist-test-slave-dt7p"
server_key: "f92b91bc543b74679d3885952c257bf1"
server_key_iv: "2ebd51cc8959a345af3eadc996b93f71"
server_key_version: "kuduclusterkey@0"
I20260926 03:34:27.025125 12929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-1/wal/instance:
uuid: "bdef2b583f0640108633f2970851619c"
format_stamp: "Formatted at 2026-09-26 03:34:27 on dist-test-slave-dt7p"
server_key: "f92b91bc543b74679d3885952c257bf1"
server_key_iv: "2ebd51cc8959a345af3eadc996b93f71"
server_key_version: "kuduclusterkey@0"
I20260926 03:34:27.033341 12929 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260926 03:34:27.055433 12953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:27.057410 12929 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 03:34:27.057643 12929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-1/wal
uuid: "bdef2b583f0640108633f2970851619c"
format_stamp: "Formatted at 2026-09-26 03:34:27 on dist-test-slave-dt7p"
server_key: "f92b91bc543b74679d3885952c257bf1"
server_key_iv: "2ebd51cc8959a345af3eadc996b93f71"
server_key_version: "kuduclusterkey@0"
I20260926 03:34:27.057926 12929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:34:27.076486 12929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:34:27.081420 12929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:34:27.081773 12929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:34:27.083045 12929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:34:27.084832 12929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:34:27.084952 12929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:27.085119 12929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:34:27.085202 12929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:27.139292 12929 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34493
I20260926 03:34:27.139318 13065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34493 every 8 connection(s)
I20260926 03:34:27.141115 12929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:34:27.141742 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 12929
I20260926 03:34:27.141993 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:34:27.142848 388 external_mini_cluster.cc:1428] Setting key d301bb967e115e4db712afbf060f51db
Sep 26 03:34:27 dist-test-slave-dt7p krb5kdc[12234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 03:34:27.188246 13067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:27.147285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43545 (local address 127.0.97.62:37003)
0926 03:34:27.167231 (+ 19946us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:27.167237 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:27.168079 (+ 842us) server_negotiation.cc:407] Connection header received
0926 03:34:27.170629 (+ 2550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:27.170636 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:27.170751 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:27.170893 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:27.172869 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:27.173715 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:27.175427 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:27.175884 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:27.176030 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:27.180300 (+ 4270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:27.180309 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:27.180317 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:27.180382 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:27.184610 (+ 4228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:27.185470 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:27.185476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:27.185483 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:27.185668 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:27.186114 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:27.186119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:27.186123 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:27.186566 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:27.186755 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:27.187665 (+ 910us) server_negotiation.cc:299] Negotiation successful
0926 03:34:27.187906 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19766,"thread_start_us":21330,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 03:34:27.189738 13068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:27.148208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37003 (local address 127.0.97.2:43545)
0926 03:34:27.167839 (+ 19631us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:27.167893 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 03:34:27.170255 (+ 2362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:34:27.171199 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:34:27.171211 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:34:27.171736 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:34:27.172653 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:27.172676 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:27.173880 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:27.173886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:27.175255 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:27.175272 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:27.176135 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:27.176142 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:27.176412 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:27.177274 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:34:27.177301 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:34:27.180019 (+ 2718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:34:27.184779 (+ 4760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:27.184789 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:27.184807 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:27.185317 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:27.185797 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:27.185800 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:27.185802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:27.185998 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:27.186718 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:34:27.186726 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:34:27.187139 (+ 413us) client_negotiation.cc:769] Sending connection context
0926 03:34:27.187426 (+ 287us) client_negotiation.cc:240] Negotiation successful
0926 03:34:27.188041 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19291,"thread_start_us":19056,"threads_started":1}
I20260926 03:34:27.191712 13066 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37003
I20260926 03:34:27.192152 13066 heartbeater.cc:464] Registering TS with master...
I20260926 03:34:27.193233 13066 heartbeater.cc:511] Master 127.0.97.62:37003 requested a full tablet report, sending...
I20260926 03:34:27.195192 12710 ts_manager.cc:194] Registered new tserver with Master: bdef2b583f0640108633f2970851619c (127.0.97.2:34493)
I20260926 03:34:27.196151 12710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43545
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:34:27.224218 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:60423/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37003
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:34:27.297987 12923 heartbeater.cc:503] Master 127.0.97.62:37003 was elected leader, sending a full tablet report...
W20260926 03:34:27.614768 13072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:34:27.615123 13072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:34:27.615226 13072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:34:27.623211 13072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:34:27.623358 13072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:34:27.623485 13072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:34:27.634533 13072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:60423/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:34:27.636204 13072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:34:27.637770 13072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:34:27.653136 13077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:27.653133 13078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:27.653122 13080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:34:27.658488 13072 server_base.cc:1034] running on GCE node
I20260926 03:34:27.659299 13072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:34:27.660586 13072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:34:27.661834 13072 hybrid_clock.cc:648] HybridClock initialized: now 1790393667661756 us; error 64 us; skew 500 ppm
I20260926 03:34:27.662267 13072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:34:27 dist-test-slave-dt7p krb5kdc[12234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393667, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:34:27.666214 13072 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:34:27.667968 13072 webserver.cc:460] Webserver started at http://127.0.97.3:45677/ using document root <none> and password file <none>
I20260926 03:34:27.668910 13072 fs_manager.cc:359] Metadata directory not provided
I20260926 03:34:27.669049 13072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:34:27.669430 13072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:34:27 dist-test-slave-dt7p krb5kdc[12234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393667, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:34:27.686993 13072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-2/data/instance:
uuid: "94b432392997422fa6a7077c3856a735"
format_stamp: "Formatted at 2026-09-26 03:34:27 on dist-test-slave-dt7p"
server_key: "c42184c2dcb01c399dc856205b2d2cf1"
server_key_iv: "6ca723acf4c47790dc7d3411ea1b08b7"
server_key_version: "kuduclusterkey@0"
I20260926 03:34:27.688050 13072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-2/wal/instance:
uuid: "94b432392997422fa6a7077c3856a735"
format_stamp: "Formatted at 2026-09-26 03:34:27 on dist-test-slave-dt7p"
server_key: "c42184c2dcb01c399dc856205b2d2cf1"
server_key_iv: "6ca723acf4c47790dc7d3411ea1b08b7"
server_key_version: "kuduclusterkey@0"
I20260926 03:34:27.694846 13072 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260926 03:34:27.713871 13087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:27.715781 13072 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 03:34:27.716005 13072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-2/wal
uuid: "94b432392997422fa6a7077c3856a735"
format_stamp: "Formatted at 2026-09-26 03:34:27 on dist-test-slave-dt7p"
server_key: "c42184c2dcb01c399dc856205b2d2cf1"
server_key_iv: "6ca723acf4c47790dc7d3411ea1b08b7"
server_key_version: "kuduclusterkey@0"
I20260926 03:34:27.716280 13072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:34:27.739715 13072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:34:27.744320 13072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:34:27.744733 13072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:34:27.745891 13072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:34:27.747423 13072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:34:27.747503 13072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:27.747651 13072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:34:27.747704 13072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:27.785223 13072 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41181
I20260926 03:34:27.785254 13199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41181 every 8 connection(s)
I20260926 03:34:27.786813 13072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:34:27.793555 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 13072
I20260926 03:34:27.793767 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:34:27.794406 388 external_mini_cluster.cc:1428] Setting key ee0baee8f69a3613b7e27c0a710706db
Sep 26 03:34:27 dist-test-slave-dt7p krb5kdc[12234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393667, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:34:27.809532 13202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:27.790142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59157 (local address 127.0.97.62:37003)
0926 03:34:27.790757 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:27.790762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:27.791775 (+ 1013us) server_negotiation.cc:407] Connection header received
0926 03:34:27.794150 (+ 2375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:27.794155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:27.794253 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:27.794360 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:27.796090 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:27.797270 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:27.799119 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:27.799576 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:27.799676 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:27.803220 (+ 3544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:27.803225 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:27.803229 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:27.803263 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:27.806145 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:27.806929 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:27.806933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:27.806936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:27.807107 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:27.807514 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:27.807517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:27.807519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:27.807799 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:27.807914 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:27.808837 (+ 923us) server_negotiation.cc:299] Negotiation successful
0926 03:34:27.809107 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":346,"threads_started":1}
I20260926 03:34:27.810910 13203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:27.790783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37003 (local address 127.0.97.3:59157)
0926 03:34:27.791593 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:27.791634 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 03:34:27.793825 (+ 2191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:34:27.794668 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:34:27.794692 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:34:27.795257 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:34:27.795906 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:27.795924 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:27.797425 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:27.797429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:27.798995 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:27.799006 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:27.799777 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:27.799781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:27.800002 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:27.800773 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:34:27.800799 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:34:27.802995 (+ 2196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:34:27.806301 (+ 3306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:27.806309 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:27.806323 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:27.806790 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:27.807201 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:27.807204 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:27.807206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:27.807423 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:27.807903 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:34:27.807909 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:34:27.808340 (+ 431us) client_negotiation.cc:769] Sending connection context
0926 03:34:27.808770 (+ 430us) client_negotiation.cc:240] Negotiation successful
0926 03:34:27.809363 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":332,"threads_started":1}
I20260926 03:34:27.812489 13200 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37003
I20260926 03:34:27.812853 13200 heartbeater.cc:464] Registering TS with master...
I20260926 03:34:27.813691 13200 heartbeater.cc:511] Master 127.0.97.62:37003 requested a full tablet report, sending...
I20260926 03:34:27.815207 12710 ts_manager.cc:194] Registered new tserver with Master: 94b432392997422fa6a7077c3856a735 (127.0.97.3:41181)
I20260926 03:34:27.816082 12710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59157
I20260926 03:34:27.822793 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:34:27.824716 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 12790
Sep 26 03:34:27 dist-test-slave-dt7p krb5kdc[12234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:34:27.842628 13205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:27.825675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46142 (local address 127.0.97.1:40605)
0926 03:34:27.826387 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:27.826395 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:27.826424 (+ 29us) server_negotiation.cc:407] Connection header received
0926 03:34:27.826496 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:27.826503 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:27.826738 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:27.826880 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:27.828397 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:27.829589 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:27.831145 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:27.831762 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:27.831873 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:27.836522 (+ 4649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:27.836535 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:27.836552 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:27.836662 (+ 110us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:27.839733 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:27.840400 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:27.840407 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:27.840410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:27.840617 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:27.840989 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:27.840996 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:27.841001 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:27.841501 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:27.841791 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:27.842048 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 03:34:27.842315 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":379,"threads_started":1}
I20260926 03:34:27.844065 12918 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:34:27.938593 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 12790
I20260926 03:34:27.956363 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 12929
Sep 26 03:34:27 dist-test-slave-dt7p krb5kdc[12234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:34:27.975850 13207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:27.957508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49738 (local address 127.0.97.2:34493)
0926 03:34:27.958217 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:27.958226 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:27.958260 (+ 34us) server_negotiation.cc:407] Connection header received
0926 03:34:27.958339 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:27.958348 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:27.958635 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:27.958788 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:27.960162 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:27.961555 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:27.963061 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:27.963731 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:27.963852 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:27.969039 (+ 5187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:27.969053 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:27.969072 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:27.969116 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:27.972203 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:27.973181 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:27.973191 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:27.973194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:27.973377 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:27.973860 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:27.973865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:27.973867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:27.974395 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:27.974692 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:27.975111 (+ 419us) server_negotiation.cc:299] Negotiation successful
0926 03:34:27.975464 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":370,"threads_started":1}
I20260926 03:34:27.977542 13061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:34:28.066816 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 12929
I20260926 03:34:28.084509 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 13072
Sep 26 03:34:28 dist-test-slave-dt7p krb5kdc[12234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:34:28.109675 13209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:28.085765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46148 (local address 127.0.97.3:41181)
0926 03:34:28.086467 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:28.086476 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:28.086510 (+ 34us) server_negotiation.cc:407] Connection header received
0926 03:34:28.086585 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:28.086593 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:28.086898 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:28.087045 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:28.088447 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:28.089591 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:28.090972 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:28.091632 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:28.091785 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:28.102768 (+ 10983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:28.102787 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:28.102809 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:28.102862 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:28.106373 (+ 3511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:28.107232 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:28.107242 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:28.107244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:28.107389 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:28.107802 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:28.107807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:28.107810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:28.108235 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:28.108616 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:28.108950 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 03:34:28.109250 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":356,"threads_started":1}
I20260926 03:34:28.111474 13195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:34:28.201004 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 13072
I20260926 03:34:28.219053 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 12678
I20260926 03:34:28.220530 12740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:34:28.342862 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 12678
I20260926 03:34:28.369338 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37003
--webserver_interface=127.0.97.62
--webserver_port=33765
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37003
--ranger_config_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:60423/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:60423/kms with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:34:28.748461 13212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:34:28.748951 13212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:34:28.749133 13212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:34:28.757599 13212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:34:28.757797 13212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:34:28.757925 13212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:34:28.758038 13212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:34:28.758136 13212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:34:28.770008 13212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:60423/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37003
--ranger_config_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=33765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:34:28.771767 13212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:34:28.773489 13212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:34:28.787421 13217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:28.787345 13220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:28.787453 13218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:34:28.788681 13212 server_base.cc:1034] running on GCE node
I20260926 03:34:28.790185 13212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:34:28.792088 13212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:34:28.793330 13212 hybrid_clock.cc:648] HybridClock initialized: now 1790393668793264 us; error 53 us; skew 500 ppm
I20260926 03:34:28.793732 13212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:34:28 dist-test-slave-dt7p krb5kdc[12234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:34:28.797442 13212 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:34:28.799311 13212 webserver.cc:460] Webserver started at http://127.0.97.62:33765/ using document root <none> and password file <none>
I20260926 03:34:28.800113 13212 fs_manager.cc:359] Metadata directory not provided
I20260926 03:34:28.800231 13212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 03:34:28 dist-test-slave-dt7p krb5kdc[12234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:34:28.856062 13212 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:34:28.859563 13228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:28.861171 13212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:34:28.861397 13212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "7f2eb46af1d04155923f8023f100f9a0"
format_stamp: "Formatted at 2026-09-26 03:34:24 on dist-test-slave-dt7p"
server_key: "33f3b913d42b9a9164bb638495f24303"
server_key_iv: "9a39c75e385bc4f24775ea6236f793f7"
server_key_version: "kuduclusterkey@0"
I20260926 03:34:28.862553 13212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:34:28.879860 13212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:34:28.884311 13212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:34:28.884713 13212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:34:28.904618 13212 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37003
I20260926 03:34:28.904675 13279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37003 every 8 connection(s)
I20260926 03:34:28.906338 13212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:34:28.908140 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 13212
I20260926 03:34:28.909121 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:60423/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:40605
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:37003
--webserver_port=40201
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:60423/kms with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:34:28.914106 13280 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:34:28.917915 13280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0: Bootstrap starting.
I20260926 03:34:28.944962 13280 log.cc:824] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0: Log is configured to *not* fsync() on all Append() calls
I20260926 03:34:28.955287 13280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:34:28.955910 13280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0: Bootstrap complete.
I20260926 03:34:28.967994 13280 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f2eb46af1d04155923f8023f100f9a0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37003 } }
I20260926 03:34:28.969288 13280 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f2eb46af1d04155923f8023f100f9a0, State: Initialized, Role: FOLLOWER
I20260926 03:34:28.970135 13280 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f2eb46af1d04155923f8023f100f9a0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37003 } }
I20260926 03:34:28.970418 13280 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:34:28.970556 13280 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:34:28.970757 13280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0 [term 1 FOLLOWER]: Advancing to term 2
I20260926 03:34:28.975307 13280 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f2eb46af1d04155923f8023f100f9a0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37003 } }
I20260926 03:34:28.975888 13280 leader_election.cc:302] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f2eb46af1d04155923f8023f100f9a0; no voters:
I20260926 03:34:28.976629 13280 leader_election.cc:288] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 03:34:28.976814 13284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 03:34:28.977926 13284 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0 [term 2 LEADER]: Becoming Leader. State: Replica: 7f2eb46af1d04155923f8023f100f9a0, State: Running, Role: LEADER
I20260926 03:34:28.978639 13284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f2eb46af1d04155923f8023f100f9a0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37003 } }
I20260926 03:34:28.979195 13280 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:34:28.982033 13286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f2eb46af1d04155923f8023f100f9a0. Latest consensus state: current_term: 2 leader_uuid: "7f2eb46af1d04155923f8023f100f9a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f2eb46af1d04155923f8023f100f9a0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37003 } } }
I20260926 03:34:28.982515 13286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:34:28.983175 13285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7f2eb46af1d04155923f8023f100f9a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f2eb46af1d04155923f8023f100f9a0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37003 } } }
I20260926 03:34:28.983506 13285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:34:29.004554 13290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:34:29.018739 13290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:34:29.020968 13290 catalog_manager.cc:1259] Loaded cluster ID: 2746431caede439b81ca2dab88174f8d
I20260926 03:34:29.021216 13290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:34:29.026615 13290 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:34:29.028292 13290 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0: Loaded TSK: 0
I20260926 03:34:29.030702 13290 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:34:29.038874 13280 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260926 03:34:29.406759 13282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:34:29.407424 13282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:34:29.407652 13282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:34:29.419981 13282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:34:29.420204 13282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:34:29.420476 13282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:34:29.437641 13282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:60423/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:40605
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=40201
--tserver_master_addrs=127.0.97.62:37003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:34:29.440299 13282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:34:29.442674 13282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:34:29.481074 13319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:29.485277 13318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:29.488287 13321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:34:29.488732 13282 server_base.cc:1034] running on GCE node
I20260926 03:34:29.489706 13282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:34:29.492033 13282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:34:29.493499 13282 hybrid_clock.cc:648] HybridClock initialized: now 1790393669493430 us; error 48 us; skew 500 ppm
I20260926 03:34:29.493996 13282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:34:29 dist-test-slave-dt7p krb5kdc[12234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:34:29.499143 13282 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:34:29.518927 13282 webserver.cc:460] Webserver started at http://127.0.97.1:40201/ using document root <none> and password file <none>
I20260926 03:34:29.520347 13282 fs_manager.cc:359] Metadata directory not provided
I20260926 03:34:29.520596 13282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 03:34:29 dist-test-slave-dt7p krb5kdc[12234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:34:29.564128 13282 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.005s sys 0.000s
I20260926 03:34:29.574880 13329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:29.577054 13282 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.008s sys 0.001s
I20260926 03:34:29.577373 13282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-0/wal
uuid: "e2a557024c9b420a975c3abf0f64a40c"
format_stamp: "Formatted at 2026-09-26 03:34:25 on dist-test-slave-dt7p"
server_key: "5a4eb17e59153422b08c4cc1abd9673b"
server_key_iv: "e0146bc238645bcd042575b9c3a92e3f"
server_key_version: "kuduclusterkey@0"
I20260926 03:34:29.579101 13282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:34:29.631217 13282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:34:29.637010 13282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:34:29.637435 13282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:34:29.645063 13282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:34:29.647064 13282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:34:29.647208 13282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:29.647378 13282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:34:29.647485 13282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:29.733453 13282 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40605
I20260926 03:34:29.733526 13442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40605 every 8 connection(s)
I20260926 03:34:29.735492 13282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:34:29.745628 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 13282
I20260926 03:34:29.746430 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:60423/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:34493
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:37003
--webserver_port=36379
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:60423/kms with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:34:29 dist-test-slave-dt7p krb5kdc[12234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:34:29.774684 13446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:29.739692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37003 (local address 127.0.97.1:59231)
0926 03:34:29.740831 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:29.740871 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 03:34:29.743505 (+ 2634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:34:29.746009 (+ 2504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:34:29.746020 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:34:29.746723 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:34:29.747698 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:29.747727 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:29.750567 (+ 2840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:29.750576 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:29.752978 (+ 2402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:29.752997 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:29.757415 (+ 4418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:29.757425 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:29.757768 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:29.758824 (+ 1056us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:34:29.758864 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:34:29.762122 (+ 3258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:34:29.767756 (+ 5634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:29.767772 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:29.767797 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:29.768598 (+ 801us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:29.769253 (+ 655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:29.769259 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:29.769263 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:29.769588 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:29.770827 (+ 1239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:34:29.770840 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:34:29.771394 (+ 554us) client_negotiation.cc:769] Sending connection context
0926 03:34:29.771798 (+ 404us) client_negotiation.cc:240] Negotiation successful
0926 03:34:29.772641 (+ 843us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"thread_start_us":604,"threads_started":1}
I20260926 03:34:29.778134 13445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:29.739489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59231 (local address 127.0.97.62:37003)
0926 03:34:29.740463 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:29.740486 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:29.741037 (+ 551us) server_negotiation.cc:407] Connection header received
0926 03:34:29.744447 (+ 3410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:29.744462 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:29.745016 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:29.745476 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:29.748136 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:29.750316 (+ 2180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:29.753188 (+ 2872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:29.756668 (+ 3480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:29.756897 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:29.762487 (+ 5590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:29.762501 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:29.762530 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:29.762598 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:29.767487 (+ 4889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:29.768806 (+ 1319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:29.768819 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:29.768830 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:29.769080 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:29.769754 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:29.769760 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:29.769764 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:29.770626 (+ 862us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:29.770878 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:29.775795 (+ 4917us) server_negotiation.cc:299] Negotiation successful
0926 03:34:29.776363 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":365,"threads_started":1}
I20260926 03:34:29.782392 13443 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37003
I20260926 03:34:29.782779 13443 heartbeater.cc:464] Registering TS with master...
I20260926 03:34:29.783857 13443 heartbeater.cc:511] Master 127.0.97.62:37003 requested a full tablet report, sending...
W20260926 03:34:29.788172 13443 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37003 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 03:34:29.788414 13443 heartbeater.cc:464] Registering TS with master...
I20260926 03:34:29.788769 13443 heartbeater.cc:511] Master 127.0.97.62:37003 requested a full tablet report, sending...
I20260926 03:34:29.790122 13443 heartbeater.cc:464] Registering TS with master...
I20260926 03:34:29.790463 13443 heartbeater.cc:511] Master 127.0.97.62:37003 requested a full tablet report, sending...
W20260926 03:34:29.791549 13443 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 03:34:30.201990 13447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:34:30.202339 13447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:34:30.202514 13447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:34:30.211377 13447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:34:30.211545 13447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:34:30.211745 13447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:34:30.222644 13447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:60423/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:34493
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=36379
--tserver_master_addrs=127.0.97.62:37003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:34:30.224357 13447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:34:30.225989 13447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:34:30.243305 13461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:30.245225 13462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:34:30.253031 13447 server_base.cc:1034] running on GCE node
W20260926 03:34:30.253490 13464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:34:30.254487 13447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:34:30.264902 13447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:34:30.266863 13447 hybrid_clock.cc:648] HybridClock initialized: now 1790393670266776 us; error 88 us; skew 500 ppm
I20260926 03:34:30.267361 13447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:34:30 dist-test-slave-dt7p krb5kdc[12234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:34:30.272428 13447 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:34:30.281960 13447 webserver.cc:460] Webserver started at http://127.0.97.2:36379/ using document root <none> and password file <none>
I20260926 03:34:30.283207 13447 fs_manager.cc:359] Metadata directory not provided
I20260926 03:34:30.283398 13447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 03:34:30 dist-test-slave-dt7p krb5kdc[12234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:34:30.319440 13447 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.008s sys 0.004s
I20260926 03:34:30.328866 13474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:30.331385 13447 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.001s sys 0.006s
I20260926 03:34:30.331645 13447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-1/wal
uuid: "bdef2b583f0640108633f2970851619c"
format_stamp: "Formatted at 2026-09-26 03:34:27 on dist-test-slave-dt7p"
server_key: "f92b91bc543b74679d3885952c257bf1"
server_key_iv: "2ebd51cc8959a345af3eadc996b93f71"
server_key_version: "kuduclusterkey@0"
I20260926 03:34:30.333467 13447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:34:30.369692 13447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
Sep 26 03:34:30 dist-test-slave-dt7p krb5kdc[12234](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790393670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:34:30.377789 13447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:34:30.378245 13447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:34:30.379844 13447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:34:30.383280 13447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:34:30.383410 13447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:30.383565 13447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:34:30.383661 13447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:30.480778 13447 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34493
I20260926 03:34:30.480804 13587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34493 every 8 connection(s)
I20260926 03:34:30.482647 13447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:34:30.488452 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 13447
I20260926 03:34:30.489496 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:60423/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:41181
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:37003
--webserver_port=45677
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:60423/kms with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:34:30 dist-test-slave-dt7p krb5kdc[12234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:34:30.536279 13591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:30.488059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50645 (local address 127.0.97.62:37003)
0926 03:34:30.506666 (+ 18607us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:30.506673 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:30.506704 (+ 31us) server_negotiation.cc:407] Connection header received
0926 03:34:30.511981 (+ 5277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:30.511989 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:30.512119 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:30.512225 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:30.515032 (+ 2807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:30.516180 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:30.518731 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:30.519476 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:30.519615 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:30.525816 (+ 6201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:30.525828 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:30.525834 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:30.525899 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:30.531250 (+ 5351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:30.532910 (+ 1660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:30.532917 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:30.532921 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:30.533095 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:30.533808 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:30.533813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:30.533817 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:30.534236 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:30.534386 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:30.535544 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0926 03:34:30.535828 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18448,"thread_start_us":21270,"threads_started":1}
I20260926 03:34:30.539533 13590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:30.487295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37003 (local address 127.0.97.2:50645)
0926 03:34:30.505594 (+ 18299us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:30.505673 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0926 03:34:30.511491 (+ 5818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:34:30.513006 (+ 1515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:34:30.513021 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:34:30.513751 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:34:30.514773 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:30.514802 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:30.516412 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:30.516417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:30.518537 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:30.518553 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:30.520141 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:30.520154 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:30.520612 (+ 458us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:30.521672 (+ 1060us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:34:30.521713 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:34:30.525437 (+ 3724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:34:30.531852 (+ 6415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:30.531864 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:30.531890 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:30.532747 (+ 857us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:30.533250 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:30.533254 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:30.533259 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:30.533582 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:30.534486 (+ 904us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:34:30.534495 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:34:30.535038 (+ 543us) client_negotiation.cc:769] Sending connection context
0926 03:34:30.536943 (+ 1905us) client_negotiation.cc:240] Negotiation successful
0926 03:34:30.537751 (+ 808us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17885,"thread_start_us":17558,"threads_started":1}
I20260926 03:34:30.541951 13588 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37003
I20260926 03:34:30.542337 13588 heartbeater.cc:464] Registering TS with master...
I20260926 03:34:30.543344 13588 heartbeater.cc:511] Master 127.0.97.62:37003 requested a full tablet report, sending...
I20260926 03:34:30.545699 13245 ts_manager.cc:194] Registered new tserver with Master: bdef2b583f0640108633f2970851619c (127.0.97.2:34493)
I20260926 03:34:30.548070 13245 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50645
I20260926 03:34:30.791805 13443 heartbeater.cc:464] Registering TS with master...
I20260926 03:34:30.792455 13443 heartbeater.cc:511] Master 127.0.97.62:37003 requested a full tablet report, sending...
I20260926 03:34:30.794373 13245 ts_manager.cc:194] Registered new tserver with Master: e2a557024c9b420a975c3abf0f64a40c (127.0.97.1:40605)
I20260926 03:34:30.795765 13245 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59231
Sep 26 03:34:30 dist-test-slave-dt7p krb5kdc[12234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260926 03:34:31.029740 13592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:34:31.030306 13592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:34:31.030484 13592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:34:31.043457 13592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:34:31.043696 13592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:34:31.043918 13592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:34:31.060252 13592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:60423/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:41181
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=45677
--tserver_master_addrs=127.0.97.62:37003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:34:31.062547 13592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:34:31.064522 13592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:34:31.083881 13598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:31.084949 13597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:31.092849 13600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:34:31.093246 13592 server_base.cc:1034] running on GCE node
I20260926 03:34:31.094098 13592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:34:31.098752 13592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:34:31.100100 13592 hybrid_clock.cc:648] HybridClock initialized: now 1790393671100024 us; error 65 us; skew 500 ppm
I20260926 03:34:31.100632 13592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:34:31 dist-test-slave-dt7p krb5kdc[12234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:34:31.105453 13592 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:34:31.121932 13592 webserver.cc:460] Webserver started at http://127.0.97.3:45677/ using document root <none> and password file <none>
I20260926 03:34:31.123232 13592 fs_manager.cc:359] Metadata directory not provided
I20260926 03:34:31.123493 13592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 03:34:31 dist-test-slave-dt7p krb5kdc[12234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 26 03:34:31 dist-test-slave-dt7p krb5kdc[12234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:34:31.163666 13592 fs_manager.cc:709] Time spent opening directory manager: real 0.015s user 0.009s sys 0.000s
I20260926 03:34:31.188076 13609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:31.191156 13592 fs_manager.cc:725] Time spent opening block manager: real 0.025s user 0.010s sys 0.012s
I20260926 03:34:31.191511 13592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-2/wal
uuid: "94b432392997422fa6a7077c3856a735"
format_stamp: "Formatted at 2026-09-26 03:34:27 on dist-test-slave-dt7p"
server_key: "c42184c2dcb01c399dc856205b2d2cf1"
server_key_iv: "6ca723acf4c47790dc7d3411ea1b08b7"
server_key_version: "kuduclusterkey@0"
I20260926 03:34:31.193132 13592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:34:31.275321 13592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:34:31.281144 13592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:34:31.281531 13592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:34:31.282925 13592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:34:31.284926 13592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:34:31.285084 13592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:31.285243 13592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:34:31.285378 13592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:31.410389 13723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41181 every 8 connection(s)
I20260926 03:34:31.410485 13592 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41181
I20260926 03:34:31.412968 13592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:34:31.421463 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 13592
Sep 26 03:34:31 dist-test-slave-dt7p krb5kdc[12234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:34:31.454487 13728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:31.418300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52321 (local address 127.0.97.62:37003)
0926 03:34:31.430006 (+ 11706us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:31.430017 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:31.430049 (+ 32us) server_negotiation.cc:407] Connection header received
0926 03:34:31.430151 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:31.430160 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:31.430300 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:31.433527 (+ 3227us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:31.433801 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:31.435339 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:31.438658 (+ 3319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:31.439389 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:31.439525 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:31.446020 (+ 6495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:31.446028 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:31.446033 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:31.446099 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:31.450116 (+ 4017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:31.451313 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:31.451318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:31.451322 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:31.451477 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:31.452132 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:31.452140 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:31.452143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:31.452583 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:31.452751 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:31.453734 (+ 983us) server_negotiation.cc:299] Negotiation successful
0926 03:34:31.454037 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11496,"thread_start_us":12132,"threads_started":1}
I20260926 03:34:31.456032 13727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:31.417476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37003 (local address 127.0.97.3:52321)
0926 03:34:31.418532 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:31.418591 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 03:34:31.421884 (+ 3293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:34:31.431035 (+ 9151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:34:31.431057 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:34:31.431996 (+ 939us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:34:31.433259 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:31.433295 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:31.435580 (+ 2285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:31.435588 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:31.438411 (+ 2823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:31.438449 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:31.439957 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:31.439967 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:31.440454 (+ 487us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:31.441853 (+ 1399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:34:31.441895 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:34:31.445655 (+ 3760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:34:31.450372 (+ 4717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:31.450383 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:31.450413 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:31.451123 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:31.451613 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:31.451618 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:31.451621 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:31.451954 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:31.452734 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:34:31.452747 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:34:31.453295 (+ 548us) client_negotiation.cc:769] Sending connection context
0926 03:34:31.453636 (+ 341us) client_negotiation.cc:240] Negotiation successful
0926 03:34:31.454320 (+ 684us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":859,"threads_started":1}
I20260926 03:34:31.458441 13724 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37003
I20260926 03:34:31.458847 13724 heartbeater.cc:464] Registering TS with master...
I20260926 03:34:31.459872 13724 heartbeater.cc:511] Master 127.0.97.62:37003 requested a full tablet report, sending...
I20260926 03:34:31.461705 13245 ts_manager.cc:194] Registered new tserver with Master: 94b432392997422fa6a7077c3856a735 (127.0.97.3:41181)
I20260926 03:34:31.463084 13245 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52321
I20260926 03:34:31.466745 13730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:31.423420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57664 (local address 127.0.97.62:37003)
0926 03:34:31.447931 (+ 24511us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:31.447940 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:31.448788 (+ 848us) server_negotiation.cc:407] Connection header received
0926 03:34:31.448999 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:31.449005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:31.449135 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:31.449258 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:31.451929 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:31.453173 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:31.455290 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:31.455917 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:31.456067 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:31.459797 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:31.459805 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:31.459810 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:31.459855 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:31.463606 (+ 3751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:31.464473 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:31.464479 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:31.464482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:31.464677 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:31.465093 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:31.465100 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:31.465104 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:31.465387 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:31.465689 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:31.466028 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 03:34:31.466269 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24237,"thread_start_us":24058,"threads_started":1}
I20260926 03:34:31.469774 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:34:31.490754 13730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:31.473411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57670 (local address 127.0.97.62:37003)
0926 03:34:31.473683 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:31.473693 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:31.474620 (+ 927us) server_negotiation.cc:407] Connection header received
0926 03:34:31.474922 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:31.474928 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:31.475011 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:31.475108 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:31.477213 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:31.478040 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:31.480153 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:31.480894 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:31.481037 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:31.484526 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:31.484532 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:31.484537 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:31.484611 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:31.487941 (+ 3330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:31.488786 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:31.488790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:31.488792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:31.488943 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:31.489367 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:31.489370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:31.489372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:31.489652 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:31.489888 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:31.490255 (+ 367us) server_negotiation.cc:299] Negotiation successful
0926 03:34:31.490445 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:34:31.496119 13245 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:34:31.504706 13245 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:34:31.546008 13745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:31.521142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46152 (local address 127.0.97.1:40605)
0926 03:34:31.537002 (+ 15860us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:31.537011 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:31.537041 (+ 30us) server_negotiation.cc:407] Connection header received
0926 03:34:31.537173 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:31.537183 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:31.537560 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:31.538451 (+ 891us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:34:31.538709 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:31.540180 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:31.542982 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:31.544645 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:31.544808 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:31.545113 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:31.545400 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 03:34:31.545611 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15648,"thread_start_us":15530,"threads_started":1}
I20260926 03:34:31.546706 13747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:31.522191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46164 (local address 127.0.97.3:41181)
0926 03:34:31.536233 (+ 14042us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:31.536242 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:31.537001 (+ 759us) server_negotiation.cc:407] Connection header received
0926 03:34:31.537362 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:31.537369 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:31.537554 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:31.537787 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:34:31.538897 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:31.540201 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:31.543224 (+ 3023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:31.544701 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:31.544823 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:31.545113 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:31.546137 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0926 03:34:31.546318 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13842,"thread_start_us":13670,"threads_started":1}
I20260926 03:34:31.546715 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:31.520690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40605 (local address 127.0.0.1:46152)
0926 03:34:31.528080 (+ 7390us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:31.528155 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0926 03:34:31.528412 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:34:31.537855 (+ 9443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:34:31.537869 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:34:31.537904 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:34:31.538471 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:31.538491 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:31.540401 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:31.540409 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:31.542750 (+ 2341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:31.542763 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:31.544887 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:31.544894 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:31.545091 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:31.545160 (+ 69us) client_negotiation.cc:769] Sending connection context
0926 03:34:31.545336 (+ 176us) client_negotiation.cc:240] Negotiation successful
0926 03:34:31.545562 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":952,"mutex_wait_us":7,"thread_start_us":783,"threads_started":1}
I20260926 03:34:31.547058 13748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:31.522588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49750 (local address 127.0.97.2:34493)
0926 03:34:31.535604 (+ 13016us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:31.535621 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:31.537699 (+ 2078us) server_negotiation.cc:407] Connection header received
0926 03:34:31.537887 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:31.537894 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:31.538084 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:31.538300 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:34:31.540097 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:31.541910 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:31.544079 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:31.545373 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:31.545490 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:31.545740 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:31.546433 (+ 693us) server_negotiation.cc:299] Negotiation successful
0926 03:34:31.546616 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12690,"thread_start_us":13586,"threads_started":1}
I20260926 03:34:31.547221 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:31.521736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34493 (local address 127.0.0.1:49750)
0926 03:34:31.537543 (+ 15807us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:31.537559 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:34:31.537713 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:34:31.538283 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:34:31.538288 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:34:31.538322 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:34:31.539894 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:31.539908 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:31.542081 (+ 2173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:31.542086 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:31.543915 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:31.543926 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:31.545906 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:31.545911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:31.546247 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:31.546293 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 03:34:31.546382 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 03:34:31.546501 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15673,"mutex_wait_us":1065,"thread_start_us":14733,"threads_started":1}
I20260926 03:34:31.547350 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:31.521140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41181 (local address 127.0.0.1:46164)
0926 03:34:31.536843 (+ 15703us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:31.536866 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:34:31.537153 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:34:31.538227 (+ 1074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:34:31.538234 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:34:31.538266 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:34:31.538709 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:31.538718 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:31.540397 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:31.540403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:31.543017 (+ 2614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:31.543035 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:31.545265 (+ 2230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:31.545270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:31.545467 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:31.545501 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 03:34:31.545586 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 03:34:31.545704 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15558,"mutex_wait_us":77,"thread_start_us":16119,"threads_started":1}
I20260926 03:34:31.551651 13657 tablet_service.cc:1467] Processing CreateTablet for tablet 5d08a9c2aa0244b4b0bc325e758b11d9 (DEFAULT_TABLE table=test-table [id=657b5baf7ba54843b4fe159cafdd44f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:34:31.551666 13378 tablet_service.cc:1467] Processing CreateTablet for tablet 5d08a9c2aa0244b4b0bc325e758b11d9 (DEFAULT_TABLE table=test-table [id=657b5baf7ba54843b4fe159cafdd44f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:34:31.551676 13522 tablet_service.cc:1467] Processing CreateTablet for tablet 5d08a9c2aa0244b4b0bc325e758b11d9 (DEFAULT_TABLE table=test-table [id=657b5baf7ba54843b4fe159cafdd44f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:34:31.553211 13657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d08a9c2aa0244b4b0bc325e758b11d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:34:31.553233 13378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d08a9c2aa0244b4b0bc325e758b11d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:34:31.553256 13522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d08a9c2aa0244b4b0bc325e758b11d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:34:31.553460 13588 heartbeater.cc:503] Master 127.0.97.62:37003 was elected leader, sending a full tablet report...
I20260926 03:34:31.567255 13750 tablet_bootstrap.cc:492] T 5d08a9c2aa0244b4b0bc325e758b11d9 P 94b432392997422fa6a7077c3856a735: Bootstrap starting.
I20260926 03:34:31.572259 13750 tablet_bootstrap.cc:654] T 5d08a9c2aa0244b4b0bc325e758b11d9 P 94b432392997422fa6a7077c3856a735: Neither blocks nor log segments found. Creating new log.
I20260926 03:34:31.573567 13750 log.cc:824] T 5d08a9c2aa0244b4b0bc325e758b11d9 P 94b432392997422fa6a7077c3856a735: Log is configured to *not* fsync() on all Append() calls
I20260926 03:34:31.576493 13751 tablet_bootstrap.cc:492] T 5d08a9c2aa0244b4b0bc325e758b11d9 P bdef2b583f0640108633f2970851619c: Bootstrap starting.
I20260926 03:34:31.576783 13752 tablet_bootstrap.cc:492] T 5d08a9c2aa0244b4b0bc325e758b11d9 P e2a557024c9b420a975c3abf0f64a40c: Bootstrap starting.
I20260926 03:34:31.577455 13750 tablet_bootstrap.cc:492] T 5d08a9c2aa0244b4b0bc325e758b11d9 P 94b432392997422fa6a7077c3856a735: No bootstrap required, opened a new log
I20260926 03:34:31.577786 13750 ts_tablet_manager.cc:1397] T 5d08a9c2aa0244b4b0bc325e758b11d9 P 94b432392997422fa6a7077c3856a735: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260926 03:34:31.580084 13751 tablet_bootstrap.cc:654] T 5d08a9c2aa0244b4b0bc325e758b11d9 P bdef2b583f0640108633f2970851619c: Neither blocks nor log segments found. Creating new log.
I20260926 03:34:31.580597 13752 tablet_bootstrap.cc:654] T 5d08a9c2aa0244b4b0bc325e758b11d9 P e2a557024c9b420a975c3abf0f64a40c: Neither blocks nor log segments found. Creating new log.
I20260926 03:34:31.581422 13751 log.cc:824] T 5d08a9c2aa0244b4b0bc325e758b11d9 P bdef2b583f0640108633f2970851619c: Log is configured to *not* fsync() on all Append() calls
I20260926 03:34:31.581928 13752 log.cc:824] T 5d08a9c2aa0244b4b0bc325e758b11d9 P e2a557024c9b420a975c3abf0f64a40c: Log is configured to *not* fsync() on all Append() calls
I20260926 03:34:31.591176 13752 tablet_bootstrap.cc:492] T 5d08a9c2aa0244b4b0bc325e758b11d9 P e2a557024c9b420a975c3abf0f64a40c: No bootstrap required, opened a new log
I20260926 03:34:31.591362 13751 tablet_bootstrap.cc:492] T 5d08a9c2aa0244b4b0bc325e758b11d9 P bdef2b583f0640108633f2970851619c: No bootstrap required, opened a new log
I20260926 03:34:31.591049 13750 raft_consensus.cc:348] T 5d08a9c2aa0244b4b0bc325e758b11d9 P 94b432392997422fa6a7077c3856a735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a557024c9b420a975c3abf0f64a40c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40605 } } peers { permanent_uuid: "94b432392997422fa6a7077c3856a735" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41181 } } peers { permanent_uuid: "bdef2b583f0640108633f2970851619c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34493 } }
I20260926 03:34:31.591532 13752 ts_tablet_manager.cc:1397] T 5d08a9c2aa0244b4b0bc325e758b11d9 P e2a557024c9b420a975c3abf0f64a40c: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.004s
I20260926 03:34:31.591586 13750 raft_consensus.cc:374] T 5d08a9c2aa0244b4b0bc325e758b11d9 P 94b432392997422fa6a7077c3856a735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:34:31.591665 13751 ts_tablet_manager.cc:1397] T 5d08a9c2aa0244b4b0bc325e758b11d9 P bdef2b583f0640108633f2970851619c: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20260926 03:34:31.591714 13750 raft_consensus.cc:729] T 5d08a9c2aa0244b4b0bc325e758b11d9 P 94b432392997422fa6a7077c3856a735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94b432392997422fa6a7077c3856a735, State: Initialized, Role: FOLLOWER
I20260926 03:34:31.592494 13750 consensus_queue.cc:260] T 5d08a9c2aa0244b4b0bc325e758b11d9 P 94b432392997422fa6a7077c3856a735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a557024c9b420a975c3abf0f64a40c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40605 } } peers { permanent_uuid: "94b432392997422fa6a7077c3856a735" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41181 } } peers { permanent_uuid: "bdef2b583f0640108633f2970851619c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34493 } }
I20260926 03:34:31.597420 13724 heartbeater.cc:503] Master 127.0.97.62:37003 was elected leader, sending a full tablet report...
I20260926 03:34:31.598212 13750 ts_tablet_manager.cc:1428] T 5d08a9c2aa0244b4b0bc325e758b11d9 P 94b432392997422fa6a7077c3856a735: Time spent starting tablet: real 0.020s user 0.016s sys 0.004s
I20260926 03:34:31.600805 13752 raft_consensus.cc:348] T 5d08a9c2aa0244b4b0bc325e758b11d9 P e2a557024c9b420a975c3abf0f64a40c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a557024c9b420a975c3abf0f64a40c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40605 } } peers { permanent_uuid: "94b432392997422fa6a7077c3856a735" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41181 } } peers { permanent_uuid: "bdef2b583f0640108633f2970851619c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34493 } }
I20260926 03:34:31.601280 13752 raft_consensus.cc:374] T 5d08a9c2aa0244b4b0bc325e758b11d9 P e2a557024c9b420a975c3abf0f64a40c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:34:31.601385 13752 raft_consensus.cc:729] T 5d08a9c2aa0244b4b0bc325e758b11d9 P e2a557024c9b420a975c3abf0f64a40c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2a557024c9b420a975c3abf0f64a40c, State: Initialized, Role: FOLLOWER
I20260926 03:34:31.602134 13752 consensus_queue.cc:260] T 5d08a9c2aa0244b4b0bc325e758b11d9 P e2a557024c9b420a975c3abf0f64a40c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a557024c9b420a975c3abf0f64a40c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40605 } } peers { permanent_uuid: "94b432392997422fa6a7077c3856a735" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41181 } } peers { permanent_uuid: "bdef2b583f0640108633f2970851619c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34493 } }
I20260926 03:34:31.603340 13443 heartbeater.cc:503] Master 127.0.97.62:37003 was elected leader, sending a full tablet report...
I20260926 03:34:31.603045 13751 raft_consensus.cc:348] T 5d08a9c2aa0244b4b0bc325e758b11d9 P bdef2b583f0640108633f2970851619c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a557024c9b420a975c3abf0f64a40c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40605 } } peers { permanent_uuid: "94b432392997422fa6a7077c3856a735" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41181 } } peers { permanent_uuid: "bdef2b583f0640108633f2970851619c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34493 } }
I20260926 03:34:31.603494 13751 raft_consensus.cc:374] T 5d08a9c2aa0244b4b0bc325e758b11d9 P bdef2b583f0640108633f2970851619c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:34:31.603605 13751 raft_consensus.cc:729] T 5d08a9c2aa0244b4b0bc325e758b11d9 P bdef2b583f0640108633f2970851619c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdef2b583f0640108633f2970851619c, State: Initialized, Role: FOLLOWER
I20260926 03:34:31.604032 13752 ts_tablet_manager.cc:1428] T 5d08a9c2aa0244b4b0bc325e758b11d9 P e2a557024c9b420a975c3abf0f64a40c: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260926 03:34:31.604341 13751 consensus_queue.cc:260] T 5d08a9c2aa0244b4b0bc325e758b11d9 P bdef2b583f0640108633f2970851619c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a557024c9b420a975c3abf0f64a40c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40605 } } peers { permanent_uuid: "94b432392997422fa6a7077c3856a735" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41181 } } peers { permanent_uuid: "bdef2b583f0640108633f2970851619c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34493 } }
I20260926 03:34:31.606969 13751 ts_tablet_manager.cc:1428] T 5d08a9c2aa0244b4b0bc325e758b11d9 P bdef2b583f0640108633f2970851619c: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260926 03:34:31.645758 13758 raft_consensus.cc:482] T 5d08a9c2aa0244b4b0bc325e758b11d9 P bdef2b583f0640108633f2970851619c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:34:31.646142 13758 raft_consensus.cc:504] T 5d08a9c2aa0244b4b0bc325e758b11d9 P bdef2b583f0640108633f2970851619c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a557024c9b420a975c3abf0f64a40c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40605 } } peers { permanent_uuid: "94b432392997422fa6a7077c3856a735" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41181 } } peers { permanent_uuid: "bdef2b583f0640108633f2970851619c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34493 } }
I20260926 03:34:31.648265 13758 leader_election.cc:288] T 5d08a9c2aa0244b4b0bc325e758b11d9 P bdef2b583f0640108633f2970851619c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2a557024c9b420a975c3abf0f64a40c (127.0.97.1:40605), 94b432392997422fa6a7077c3856a735 (127.0.97.3:41181)
I20260926 03:34:31.655325 13747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:31.648767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45387 (local address 127.0.97.3:41181)
0926 03:34:31.649029 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:31.649037 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:31.649988 (+ 951us) server_negotiation.cc:407] Connection header received
0926 03:34:31.650131 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:31.650137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:31.650217 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:31.650289 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:34:31.650817 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:31.651659 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:31.653569 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:31.654354 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:31.654513 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:31.654629 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:31.654906 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 03:34:31.655041 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 03:34:31.655460 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:31.648842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41181 (local address 127.0.97.2:45387)
0926 03:34:31.649882 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:31.649901 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:34:31.650010 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:34:31.650320 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:34:31.650324 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:34:31.650342 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:34:31.650637 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:31.650644 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:31.651791 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:31.651799 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:31.653408 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:31.653419 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:31.654570 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:31.654575 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:31.654742 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:31.654767 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 03:34:31.654878 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 03:34:31.655017 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":919,"mutex_wait_us":57,"thread_start_us":489,"threads_started":1}
I20260926 03:34:31.655695 13745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:31.648767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33397 (local address 127.0.97.1:40605)
0926 03:34:31.649051 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:31.649061 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:31.649900 (+ 839us) server_negotiation.cc:407] Connection header received
0926 03:34:31.650483 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:31.650488 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:31.650564 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:31.650670 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:34:31.651227 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:31.651994 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:31.653834 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:31.654728 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:31.654895 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:31.655014 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:31.655284 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 03:34:31.655427 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260926 03:34:31.655795 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:31.648834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40605 (local address 127.0.97.2:33397)
0926 03:34:31.649779 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:31.649797 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:34:31.649986 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:34:31.650686 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:34:31.650691 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:34:31.650712 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:34:31.651073 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:31.651082 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:31.652132 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:31.652140 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:31.653680 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:31.653691 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:31.654970 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:31.654975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:31.655123 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:31.655145 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 03:34:31.655255 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 03:34:31.655374 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"mutex_wait_us":23,"thread_start_us":538,"threads_started":1}
I20260926 03:34:31.656895 13677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d08a9c2aa0244b4b0bc325e758b11d9" candidate_uuid: "bdef2b583f0640108633f2970851619c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94b432392997422fa6a7077c3856a735" is_pre_election: true
I20260926 03:34:31.656904 13398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d08a9c2aa0244b4b0bc325e758b11d9" candidate_uuid: "bdef2b583f0640108633f2970851619c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2a557024c9b420a975c3abf0f64a40c" is_pre_election: true
I20260926 03:34:31.657470 13398 raft_consensus.cc:2454] T 5d08a9c2aa0244b4b0bc325e758b11d9 P e2a557024c9b420a975c3abf0f64a40c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdef2b583f0640108633f2970851619c in term 0.
I20260926 03:34:31.657471 13677 raft_consensus.cc:2454] T 5d08a9c2aa0244b4b0bc325e758b11d9 P 94b432392997422fa6a7077c3856a735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdef2b583f0640108633f2970851619c in term 0.
I20260926 03:34:31.658239 13477 leader_election.cc:302] T 5d08a9c2aa0244b4b0bc325e758b11d9 P bdef2b583f0640108633f2970851619c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 94b432392997422fa6a7077c3856a735, bdef2b583f0640108633f2970851619c, e2a557024c9b420a975c3abf0f64a40c; no voters:
I20260926 03:34:31.658689 13758 raft_consensus.cc:2781] T 5d08a9c2aa0244b4b0bc325e758b11d9 P bdef2b583f0640108633f2970851619c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:34:31.658843 13758 raft_consensus.cc:482] T 5d08a9c2aa0244b4b0bc325e758b11d9 P bdef2b583f0640108633f2970851619c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:34:31.658973 13758 raft_consensus.cc:3037] T 5d08a9c2aa0244b4b0bc325e758b11d9 P bdef2b583f0640108633f2970851619c [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:34:31.661585 13758 raft_consensus.cc:504] T 5d08a9c2aa0244b4b0bc325e758b11d9 P bdef2b583f0640108633f2970851619c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a557024c9b420a975c3abf0f64a40c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40605 } } peers { permanent_uuid: "94b432392997422fa6a7077c3856a735" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41181 } } peers { permanent_uuid: "bdef2b583f0640108633f2970851619c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34493 } }
I20260926 03:34:31.662290 13758 leader_election.cc:288] T 5d08a9c2aa0244b4b0bc325e758b11d9 P bdef2b583f0640108633f2970851619c [CANDIDATE]: Term 1 election: Requested vote from peers e2a557024c9b420a975c3abf0f64a40c (127.0.97.1:40605), 94b432392997422fa6a7077c3856a735 (127.0.97.3:41181)
I20260926 03:34:31.662768 13398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d08a9c2aa0244b4b0bc325e758b11d9" candidate_uuid: "bdef2b583f0640108633f2970851619c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2a557024c9b420a975c3abf0f64a40c"
I20260926 03:34:31.662798 13677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d08a9c2aa0244b4b0bc325e758b11d9" candidate_uuid: "bdef2b583f0640108633f2970851619c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94b432392997422fa6a7077c3856a735"
I20260926 03:34:31.663005 13677 raft_consensus.cc:3037] T 5d08a9c2aa0244b4b0bc325e758b11d9 P 94b432392997422fa6a7077c3856a735 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:34:31.663010 13398 raft_consensus.cc:3037] T 5d08a9c2aa0244b4b0bc325e758b11d9 P e2a557024c9b420a975c3abf0f64a40c [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:34:31.665514 13677 raft_consensus.cc:2454] T 5d08a9c2aa0244b4b0bc325e758b11d9 P 94b432392997422fa6a7077c3856a735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdef2b583f0640108633f2970851619c in term 1.
I20260926 03:34:31.665544 13398 raft_consensus.cc:2454] T 5d08a9c2aa0244b4b0bc325e758b11d9 P e2a557024c9b420a975c3abf0f64a40c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdef2b583f0640108633f2970851619c in term 1.
I20260926 03:34:31.665977 13475 leader_election.cc:302] T 5d08a9c2aa0244b4b0bc325e758b11d9 P bdef2b583f0640108633f2970851619c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94b432392997422fa6a7077c3856a735, bdef2b583f0640108633f2970851619c; no voters:
I20260926 03:34:31.666291 13758 raft_consensus.cc:2781] T 5d08a9c2aa0244b4b0bc325e758b11d9 P bdef2b583f0640108633f2970851619c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:34:31.666893 13758 raft_consensus.cc:686] T 5d08a9c2aa0244b4b0bc325e758b11d9 P bdef2b583f0640108633f2970851619c [term 1 LEADER]: Becoming Leader. State: Replica: bdef2b583f0640108633f2970851619c, State: Running, Role: LEADER
I20260926 03:34:31.667515 13758 consensus_queue.cc:237] T 5d08a9c2aa0244b4b0bc325e758b11d9 P bdef2b583f0640108633f2970851619c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a557024c9b420a975c3abf0f64a40c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40605 } } peers { permanent_uuid: "94b432392997422fa6a7077c3856a735" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41181 } } peers { permanent_uuid: "bdef2b583f0640108633f2970851619c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34493 } }
I20260926 03:34:31.673358 13244 catalog_manager.cc:5495] T 5d08a9c2aa0244b4b0bc325e758b11d9 P bdef2b583f0640108633f2970851619c reported cstate change: term changed from 0 to 1, leader changed from <none> to bdef2b583f0640108633f2970851619c (127.0.97.2). New cstate: current_term: 1 leader_uuid: "bdef2b583f0640108633f2970851619c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a557024c9b420a975c3abf0f64a40c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94b432392997422fa6a7077c3856a735" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdef2b583f0640108633f2970851619c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34493 } health_report { overall_health: HEALTHY } } }
W20260926 03:34:31.692072 13725 tablet.cc:2369] T 5d08a9c2aa0244b4b0bc325e758b11d9 P 94b432392997422fa6a7077c3856a735: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:34:31.725158 13748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:31.719188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49760 (local address 127.0.97.2:34493)
0926 03:34:31.719451 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:31.719458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:31.719479 (+ 21us) server_negotiation.cc:407] Connection header received
0926 03:34:31.719741 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:31.719746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:31.719838 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:31.720008 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:34:31.720416 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:31.721298 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:31.722704 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:31.723128 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:31.723267 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:31.723785 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:34:31.724509 (+ 724us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:34:31.724713 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:31.724768 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 03:34:31.724865 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 03:34:31.735208 13398 raft_consensus.cc:1261] T 5d08a9c2aa0244b4b0bc325e758b11d9 P e2a557024c9b420a975c3abf0f64a40c [term 1 FOLLOWER]: Refusing update from remote peer bdef2b583f0640108633f2970851619c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:34:31.735316 13677 raft_consensus.cc:1261] T 5d08a9c2aa0244b4b0bc325e758b11d9 P 94b432392997422fa6a7077c3856a735 [term 1 FOLLOWER]: Refusing update from remote peer bdef2b583f0640108633f2970851619c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:34:31.736320 13761 consensus_queue.cc:1035] T 5d08a9c2aa0244b4b0bc325e758b11d9 P bdef2b583f0640108633f2970851619c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2a557024c9b420a975c3abf0f64a40c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:34:31.736750 13758 consensus_queue.cc:1035] T 5d08a9c2aa0244b4b0bc325e758b11d9 P bdef2b583f0640108633f2970851619c [LEADER]: Connected to new peer: Peer: permanent_uuid: "94b432392997422fa6a7077c3856a735" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 03:34:31.739418 13444 tablet.cc:2369] T 5d08a9c2aa0244b4b0bc325e758b11d9 P e2a557024c9b420a975c3abf0f64a40c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:34:31.747972 13764 mvcc.cc:205] Tried to move back new op lower bound from 7333452479415529472 to 7333452479154868224. Current Snapshot: MvccSnapshot[applied={T|T < 7333452479415529472}]
I20260926 03:34:31.763581 13765 mvcc.cc:205] Tried to move back new op lower bound from 7333452479415529472 to 7333452479154868224. Current Snapshot: MvccSnapshot[applied={T|T < 7333452479415529472}]
I20260926 03:34:31.771265 13244 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:34:31.771628 13244 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:34:31.775899 13244 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f2eb46af1d04155923f8023f100f9a0: Sending DeleteTablet for 3 replicas of tablet 5d08a9c2aa0244b4b0bc325e758b11d9
I20260926 03:34:31.777374 13657 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d08a9c2aa0244b4b0bc325e758b11d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:34:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46164
I20260926 03:34:31.777356 13378 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d08a9c2aa0244b4b0bc325e758b11d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:34:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46152
I20260926 03:34:31.778184 13522 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d08a9c2aa0244b4b0bc325e758b11d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:34:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49750
I20260926 03:34:31.786226 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 13282
I20260926 03:34:31.789912 13777 tablet_replica.cc:330] T 5d08a9c2aa0244b4b0bc325e758b11d9 P bdef2b583f0640108633f2970851619c: stopping tablet replica
I20260926 03:34:31.790722 13777 raft_consensus.cc:2229] T 5d08a9c2aa0244b4b0bc325e758b11d9 P bdef2b583f0640108633f2970851619c [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:34:31.791886 13777 raft_consensus.cc:2258] T 5d08a9c2aa0244b4b0bc325e758b11d9 P bdef2b583f0640108633f2970851619c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:34:31.795372 13777 ts_tablet_manager.cc:1905] T 5d08a9c2aa0244b4b0bc325e758b11d9 P bdef2b583f0640108633f2970851619c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:34:31.796241 13776 tablet_replica.cc:330] T 5d08a9c2aa0244b4b0bc325e758b11d9 P e2a557024c9b420a975c3abf0f64a40c: stopping tablet replica
I20260926 03:34:31.796526 13775 tablet_replica.cc:330] T 5d08a9c2aa0244b4b0bc325e758b11d9 P 94b432392997422fa6a7077c3856a735: stopping tablet replica
I20260926 03:34:31.797029 13776 raft_consensus.cc:2229] T 5d08a9c2aa0244b4b0bc325e758b11d9 P e2a557024c9b420a975c3abf0f64a40c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:34:31.797030 13775 raft_consensus.cc:2229] T 5d08a9c2aa0244b4b0bc325e758b11d9 P 94b432392997422fa6a7077c3856a735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:34:31.797835 13776 raft_consensus.cc:2258] T 5d08a9c2aa0244b4b0bc325e758b11d9 P e2a557024c9b420a975c3abf0f64a40c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:34:31.797832 13775 raft_consensus.cc:2258] T 5d08a9c2aa0244b4b0bc325e758b11d9 P 94b432392997422fa6a7077c3856a735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:34:31.800487 13775 ts_tablet_manager.cc:1905] T 5d08a9c2aa0244b4b0bc325e758b11d9 P 94b432392997422fa6a7077c3856a735: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:34:31.805398 13776 ts_tablet_manager.cc:1905] T 5d08a9c2aa0244b4b0bc325e758b11d9 P e2a557024c9b420a975c3abf0f64a40c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:34:31.807811 13775 ts_tablet_manager.cc:1918] T 5d08a9c2aa0244b4b0bc325e758b11d9 P 94b432392997422fa6a7077c3856a735: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:34:31.808027 13775 log.cc:1196] T 5d08a9c2aa0244b4b0bc325e758b11d9 P 94b432392997422fa6a7077c3856a735: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-2/wal/wals/5d08a9c2aa0244b4b0bc325e758b11d9
I20260926 03:34:31.808658 13775 ts_tablet_manager.cc:1939] T 5d08a9c2aa0244b4b0bc325e758b11d9 P 94b432392997422fa6a7077c3856a735: Deleting consensus metadata
I20260926 03:34:31.810546 13745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:31.788291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46156 (local address 127.0.97.1:40605)
0926 03:34:31.788509 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:31.788516 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:31.788541 (+ 25us) server_negotiation.cc:407] Connection header received
0926 03:34:31.788678 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:31.788687 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:31.788826 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:31.788974 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:31.791581 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:31.793305 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:31.796003 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:31.796735 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:31.796885 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:31.801281 (+ 4396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:31.801328 (+ 47us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:31.801350 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:31.801409 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:31.805662 (+ 4253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:31.806683 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:31.806692 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:31.806696 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:31.806899 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:31.807398 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:31.807403 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:31.807407 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:31.807974 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:31.808328 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:31.809783 (+ 1455us) server_negotiation.cc:299] Negotiation successful
0926 03:34:31.810099 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 03:34:31.810986 13229 catalog_manager.cc:4841] TS 94b432392997422fa6a7077c3856a735 (127.0.97.3:41181): tablet 5d08a9c2aa0244b4b0bc325e758b11d9 (table test-table [id=657b5baf7ba54843b4fe159cafdd44f0]) successfully deleted
I20260926 03:34:31.811465 13438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:34:31.815442 13777 ts_tablet_manager.cc:1918] T 5d08a9c2aa0244b4b0bc325e758b11d9 P bdef2b583f0640108633f2970851619c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:34:31.815675 13777 log.cc:1196] T 5d08a9c2aa0244b4b0bc325e758b11d9 P bdef2b583f0640108633f2970851619c: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-1/wal/wals/5d08a9c2aa0244b4b0bc325e758b11d9
I20260926 03:34:31.816241 13777 ts_tablet_manager.cc:1939] T 5d08a9c2aa0244b4b0bc325e758b11d9 P bdef2b583f0640108633f2970851619c: Deleting consensus metadata
I20260926 03:34:31.818423 13232 catalog_manager.cc:4841] TS bdef2b583f0640108633f2970851619c (127.0.97.2:34493): tablet 5d08a9c2aa0244b4b0bc325e758b11d9 (table test-table [id=657b5baf7ba54843b4fe159cafdd44f0]) successfully deleted
I20260926 03:34:31.916848 13776 ts_tablet_manager.cc:1918] T 5d08a9c2aa0244b4b0bc325e758b11d9 P e2a557024c9b420a975c3abf0f64a40c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:34:31.917187 13776 log.cc:1196] T 5d08a9c2aa0244b4b0bc325e758b11d9 P e2a557024c9b420a975c3abf0f64a40c: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790393463219689-388-0/minicluster-data/ts-0/wal/wals/5d08a9c2aa0244b4b0bc325e758b11d9
I20260926 03:34:31.917485 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 13282
I20260926 03:34:31.917939 13776 ts_tablet_manager.cc:1939] T 5d08a9c2aa0244b4b0bc325e758b11d9 P e2a557024c9b420a975c3abf0f64a40c: Deleting consensus metadata
W20260926 03:34:31.942255 13231 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:40605 (error 108)
I20260926 03:34:31.943899 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 13447
I20260926 03:34:31.944430 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:31.943891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40605 (local address 127.0.0.1:46166)
0926 03:34:31.944087 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:31.944228 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 03:34:31.945065 13231 catalog_manager.cc:4584] TS e2a557024c9b420a975c3abf0f64a40c (127.0.97.1:40605): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5d08a9c2aa0244b4b0bc325e758b11d9: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40605: connect: Connection refused (error 111)
I20260926 03:34:31.962718 13748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:31.945117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49766 (local address 127.0.97.2:34493)
0926 03:34:31.945335 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:31.945342 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:31.945365 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:34:31.945452 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:31.945458 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:31.945566 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:31.945693 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:31.947294 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:31.948938 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:31.950436 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:31.951191 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:31.951333 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:31.954377 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:31.954401 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:31.954429 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:31.954484 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:31.959196 (+ 4712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:31.960030 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:31.960040 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:31.960044 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:31.960247 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:31.960773 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:31.960779 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:31.960782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:31.961336 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:31.961653 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:31.961994 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 03:34:31.962263 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 03:34:31.963740 13583 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:34:31.999253 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:31.998569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40605 (local address 127.0.0.1:46180)
0926 03:34:31.998860 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:31.999012 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 03:34:32.000670 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:32.000169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40605 (local address 127.0.0.1:46188)
0926 03:34:32.000358 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:32.000452 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 03:34:32.061455 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:32.060827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40605 (local address 127.0.0.1:46204)
0926 03:34:32.061093 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:32.061224 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 03:34:32.062559 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:32.062179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40605 (local address 127.0.0.1:46212)
0926 03:34:32.062334 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:32.062417 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 03:34:32.096297 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 13447
I20260926 03:34:32.118192 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 13592
I20260926 03:34:32.132606 13747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:32.119167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46170 (local address 127.0.97.3:41181)
0926 03:34:32.119390 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:32.119396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:32.119416 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:34:32.119517 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:32.119521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:32.119604 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:32.119692 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:32.121170 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:32.121964 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:32.123237 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:32.123652 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:32.123746 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:32.126705 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:32.126721 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:32.126742 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:32.126777 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:32.129804 (+ 3027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:32.130463 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:32.130471 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:32.130473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:32.130605 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:32.130930 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:32.130933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:32.130935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:32.131353 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:32.131682 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:32.131852 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 03:34:32.132088 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 03:34:32.133464 13718 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:34:32.143356 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:32.142817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40605 (local address 127.0.0.1:46216)
0926 03:34:32.143040 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:32.143152 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 03:34:32.144498 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:32.144079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40605 (local address 127.0.0.1:46218)
0926 03:34:32.144302 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:32.144381 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260926 03:34:32.229516 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 13592
I20260926 03:34:32.249076 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 13212
I20260926 03:34:32.250492 13275 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:34:32.378153 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 13212
I20260926 03:34:32.403559 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 03:34:32.749244 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 03:34:32.749470 388 mini_ranger.cc:66] Stopping Ranger...
I20260926 03:34:33.095026 388 mini_ranger.cc:68] Stopped Ranger
2026-09-26 03:34:33.095 UTC [12264] LOG: received smart shutdown request
2026-09-26 03:34:33.099 UTC [12264] LOG: background worker "logical replication launcher" (PID 12273) exited with exit code 1
2026-09-26 03:34:33.100 UTC [12268] LOG: shutting down
2026-09-26 03:34:33.290 UTC [12264] LOG: database system is shut down
2026-09-26T03:34:33Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (58311 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:34:33 dist-test-slave-dt7p krb5kdc[13791](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:34:33 dist-test-slave-dt7p krb5kdc[13791](info): set up 2 sockets
Sep 26 03:34:33 dist-test-slave-dt7p krb5kdc[13791](info): commencing operation
krb5kdc: starting...
W20260926 03:34:34.549429 388 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:34:34 dist-test-slave-dt7p krb5kdc[13791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:34:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:34:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:34:34.829216 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45279
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45279,127.0.97.61:44607,127.0.97.60:35187
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:34:35.225004 13804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:34:35.225489 13804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:34:35.225658 13804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:34:35.234215 13804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:34:35.234431 13804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:34:35.234543 13804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:34:35.234650 13804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:34:35.234731 13804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:34:35.246567 13804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45279,127.0.97.61:44607,127.0.97.60:35187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:34:35.248420 13804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:34:35.250072 13804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:34:35.264145 13812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:35.264120 13809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:35.264230 13810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:34:35.270345 13804 server_base.cc:1034] running on GCE node
I20260926 03:34:35.271349 13804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:34:35.273276 13804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:34:35.274499 13804 hybrid_clock.cc:648] HybridClock initialized: now 1790393675274453 us; error 39 us; skew 500 ppm
I20260926 03:34:35.274943 13804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:34:35 dist-test-slave-dt7p krb5kdc[13791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:34:35.279147 13804 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:34:35.281600 13804 webserver.cc:460] Webserver started at http://127.0.97.62:36151/ using document root <none> and password file <none>
I20260926 03:34:35.282739 13804 fs_manager.cc:359] Metadata directory not provided
I20260926 03:34:35.282927 13804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:34:35.283443 13804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:34:35.287664 13804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "0676e10001d94eeaa8fcbce6c06ebaed"
format_stamp: "Formatted at 2026-09-26 03:34:35 on dist-test-slave-dt7p"
server_key: "38d0ac6ed98898558d7815a18a722931"
server_key_iv: "e5021aee401b1887fb685ab8bd968c89"
server_key_version: "encryptionkey@0"
I20260926 03:34:35.288900 13804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "0676e10001d94eeaa8fcbce6c06ebaed"
format_stamp: "Formatted at 2026-09-26 03:34:35 on dist-test-slave-dt7p"
server_key: "38d0ac6ed98898558d7815a18a722931"
server_key_iv: "e5021aee401b1887fb685ab8bd968c89"
server_key_version: "encryptionkey@0"
I20260926 03:34:35.295667 13804 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 03:34:35.299975 13819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:35.301594 13804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:34:35.301852 13804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "0676e10001d94eeaa8fcbce6c06ebaed"
format_stamp: "Formatted at 2026-09-26 03:34:35 on dist-test-slave-dt7p"
server_key: "38d0ac6ed98898558d7815a18a722931"
server_key_iv: "e5021aee401b1887fb685ab8bd968c89"
server_key_version: "encryptionkey@0"
I20260926 03:34:35.302090 13804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:34:35.324286 13804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:34:35.329627 13804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:34:35.330001 13804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:34:35.352658 13804 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45279
I20260926 03:34:35.352717 13870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45279 every 8 connection(s)
I20260926 03:34:35.354429 13804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:34:35.358261 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 13804
I20260926 03:34:35.358534 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:34:35.359241 388 external_mini_cluster.cc:1428] Setting key 12fa8644f3a2b27fa7523f8ba058031b
I20260926 03:34:35.359284 13871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:34:35.367275 13871 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45279 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Sep 26 03:34:35 dist-test-slave-dt7p krb5kdc[13791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:34:35.399255 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:35.372028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41534 (local address 127.0.97.62:45279)
0926 03:34:35.374125 (+ 2097us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:35.374136 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:35.374196 (+ 60us) server_negotiation.cc:407] Connection header received
0926 03:34:35.376014 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:35.376020 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:35.376296 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:35.376536 (+ 240us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:35.378018 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:35.379369 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:35.381963 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:35.382602 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:35.382760 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:35.389210 (+ 6450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:35.389227 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:35.389252 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:35.389310 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:35.393851 (+ 4541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:35.394722 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:35.394735 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:35.394741 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:35.394957 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:35.395381 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:35.395389 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:35.395395 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:35.396275 (+ 880us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:35.396504 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:35.397566 (+ 1062us) server_negotiation.cc:299] Negotiation successful
0926 03:34:35.397906 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1850,"spinlock_wait_cycles":35584,"thread_start_us":1601,"threads_started":1}
I20260926 03:34:35.399281 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:35.372028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45279 (local address 127.0.0.1:41534)
0926 03:34:35.373165 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:35.373227 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 03:34:35.375569 (+ 2342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:34:35.376551 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:34:35.376558 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:34:35.377406 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:34:35.377805 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:35.377825 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:35.379580 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:35.379588 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:35.381771 (+ 2183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:35.381791 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:35.382893 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:35.382902 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:35.383113 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:35.383936 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:34:35.383970 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:34:35.388877 (+ 4907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:34:35.394053 (+ 5176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:35.394065 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:35.394079 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:35.394547 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:35.395072 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:35.395075 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:35.395078 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:35.395252 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:35.396472 (+ 1220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:34:35.396485 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:34:35.396966 (+ 481us) client_negotiation.cc:769] Sending connection context
0926 03:34:35.397330 (+ 364us) client_negotiation.cc:240] Negotiation successful
0926 03:34:35.398107 (+ 777us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"mutex_wait_us":141,"thread_start_us":542,"threads_started":1}
I20260926 03:34:35.402705 13871 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44607 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:34:35.404176 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:35.403602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44607 (local address 127.0.0.1:46172)
0926 03:34:35.403841 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:35.403985 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 03:34:35.404510 13821 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44607: connect: Connection refused (error 111)
I20260926 03:34:35.406569 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:35.406047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44607 (local address 127.0.0.1:46186)
0926 03:34:35.406240 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:35.406353 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 03:34:35.406827 13871 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44607: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44607: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260926 03:34:35.424283 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:44607
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45279,127.0.97.61:44607,127.0.97.60:35187
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:34:35.456261 13871 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44607 } attempt: 1
I20260926 03:34:35.458015 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:35.457381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44607 (local address 127.0.0.1:46194)
0926 03:34:35.457660 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:35.457785 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260926 03:34:35.459318 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:35.458852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44607 (local address 127.0.0.1:46208)
0926 03:34:35.459034 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:35.459123 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 03:34:35.459553 13871 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44607: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44607: connect: Connection refused (error 111)
I20260926 03:34:35.527869 13871 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44607 } attempt: 2
I20260926 03:34:35.529601 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:35.528904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44607 (local address 127.0.0.1:46210)
0926 03:34:35.529239 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:35.529381 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260926 03:34:35.530946 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:35.530495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44607 (local address 127.0.0.1:46216)
0926 03:34:35.530674 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:35.530779 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 03:34:35.531279 13871 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44607: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44607: connect: Connection refused (error 111)
I20260926 03:34:35.622611 13871 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44607 } attempt: 3
I20260926 03:34:35.624290 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:35.623521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44607 (local address 127.0.0.1:46230)
0926 03:34:35.623900 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:35.624047 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260926 03:34:35.625666 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:35.625173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44607 (local address 127.0.0.1:46236)
0926 03:34:35.625341 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:35.625445 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 03:34:35.625931 13871 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44607: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44607: connect: Connection refused (error 111)
I20260926 03:34:35.769222 13871 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44607 } attempt: 4
I20260926 03:34:35.770785 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:35.770129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44607 (local address 127.0.0.1:46248)
0926 03:34:35.770517 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:35.770623 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227}
I20260926 03:34:35.771987 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:35.771576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44607 (local address 127.0.0.1:46250)
0926 03:34:35.771750 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:35.771857 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 03:34:35.772185 13871 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44607: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44607: connect: Connection refused (error 111)
W20260926 03:34:35.816391 13878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:34:35.816867 13878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:34:35.817034 13878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:34:35.824815 13878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:34:35.824981 13878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:34:35.825085 13878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:34:35.825210 13878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:34:35.825292 13878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:34:35.836161 13878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45279,127.0.97.61:44607,127.0.97.60:35187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:44607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:34:35.837949 13878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:34:35.839622 13878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:34:35.853219 13883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:35.853161 13886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:35.853245 13884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:34:35.853937 13878 server_base.cc:1034] running on GCE node
I20260926 03:34:35.854887 13878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:34:35.856724 13878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:34:35.857993 13878 hybrid_clock.cc:648] HybridClock initialized: now 1790393675857921 us; error 68 us; skew 500 ppm
I20260926 03:34:35.858436 13878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:34:35 dist-test-slave-dt7p krb5kdc[13791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:34:35.862380 13878 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260926 03:34:35.864305 13878 webserver.cc:460] Webserver started at http://127.0.97.61:39777/ using document root <none> and password file <none>
I20260926 03:34:35.865298 13878 fs_manager.cc:359] Metadata directory not provided
I20260926 03:34:35.865468 13878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:34:35.865854 13878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:34:35.869699 13878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-1/data/instance:
uuid: "164d9567e1db4a32a2946e2d5d76dbaf"
format_stamp: "Formatted at 2026-09-26 03:34:35 on dist-test-slave-dt7p"
server_key: "791d6595c4c3def2ee5c86239c7c9847"
server_key_iv: "75bab18b49ded1b56ce337329439c4e3"
server_key_version: "encryptionkey@0"
I20260926 03:34:35.870645 13878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-1/wal/instance:
uuid: "164d9567e1db4a32a2946e2d5d76dbaf"
format_stamp: "Formatted at 2026-09-26 03:34:35 on dist-test-slave-dt7p"
server_key: "791d6595c4c3def2ee5c86239c7c9847"
server_key_iv: "75bab18b49ded1b56ce337329439c4e3"
server_key_version: "encryptionkey@0"
I20260926 03:34:35.877269 13878 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260926 03:34:35.881285 13893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:35.882861 13878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:34:35.883068 13878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-1/wal
uuid: "164d9567e1db4a32a2946e2d5d76dbaf"
format_stamp: "Formatted at 2026-09-26 03:34:35 on dist-test-slave-dt7p"
server_key: "791d6595c4c3def2ee5c86239c7c9847"
server_key_iv: "75bab18b49ded1b56ce337329439c4e3"
server_key_version: "encryptionkey@0"
I20260926 03:34:35.883338 13878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:34:35.907173 13878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:34:35.912667 13878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:34:35.913010 13878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:34:35.934144 13878 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:44607
I20260926 03:34:35.934165 13944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:44607 every 8 connection(s)
I20260926 03:34:35.935879 13878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-1/data/info.pb
I20260926 03:34:35.940335 13945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:34:35.942114 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 13878
I20260926 03:34:35.942360 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-1/wal/instance
I20260926 03:34:35.942889 388 external_mini_cluster.cc:1428] Setting key 53374fbfeee9f4d8c476ac09b656b26d
I20260926 03:34:35.948329 13945 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45279 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:34:35 dist-test-slave-dt7p krb5kdc[13791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260926 03:34:35.972441 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:35.951577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41546 (local address 127.0.97.62:45279)
0926 03:34:35.952681 (+ 1104us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:35.952692 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:35.953540 (+ 848us) server_negotiation.cc:407] Connection header received
0926 03:34:35.955642 (+ 2102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:35.955652 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:35.955777 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:35.955936 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:35.957657 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:35.958973 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:35.960653 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:35.961442 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:35.961588 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:35.965514 (+ 3926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:35.965519 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:35.965523 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:35.965574 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:35.968777 (+ 3203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:35.969682 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:35.969689 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:35.969691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:35.969814 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:35.970357 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:35.970364 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:35.970372 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:35.970737 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:35.970926 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:35.971739 (+ 813us) server_negotiation.cc:299] Negotiation successful
0926 03:34:35.971984 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":885,"thread_start_us":713,"threads_started":1}
I20260926 03:34:35.973713 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:35.952364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45279 (local address 127.0.0.1:41546)
0926 03:34:35.953346 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:35.953397 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 03:34:35.955323 (+ 1926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:34:35.956147 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:34:35.956156 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:34:35.956825 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:34:35.957462 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:35.957480 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:35.959154 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:35.959158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:35.960486 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:35.960497 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:35.961669 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:35.961674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:35.961880 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:35.962663 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:34:35.962689 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:34:35.965230 (+ 2541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:34:35.968933 (+ 3703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:35.968943 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:35.968963 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:35.969540 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:35.969932 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:35.969938 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:35.969942 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:35.970221 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:35.970896 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:34:35.970905 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:34:35.971342 (+ 437us) client_negotiation.cc:769] Sending connection context
0926 03:34:35.971639 (+ 297us) client_negotiation.cc:240] Negotiation successful
0926 03:34:35.972235 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":436,"threads_started":1}
I20260926 03:34:35.976121 13945 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44607 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:34:35 dist-test-slave-dt7p krb5kdc[13791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 03:34:35.994131 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:35.977147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44607 (local address 127.0.0.1:46258)
0926 03:34:35.977394 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:35.977412 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:34:35.977569 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:34:35.978375 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:34:35.978380 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:34:35.978856 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:34:35.979002 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:35.979010 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:35.980918 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:35.980926 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:35.981987 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:35.981999 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:35.983329 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:35.983333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:35.983496 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:35.984173 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:34:35.984190 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:34:35.986497 (+ 2307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:34:35.991213 (+ 4716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:35.991217 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:35.991220 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:35.991634 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:35.992102 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:35.992106 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:35.992109 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:35.992274 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:35.993238 (+ 964us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:34:35.993243 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:34:35.993402 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 03:34:35.993543 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 03:34:35.993804 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 03:34:35.994696 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:35.977262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46258 (local address 127.0.97.61:44607)
0926 03:34:35.977919 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:35.977926 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:35.977952 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:34:35.977996 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:35.978002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:35.978223 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:35.978386 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:35.979212 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:35.980738 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:35.982146 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:35.983167 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:35.983312 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:35.986736 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:35.986754 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:35.986780 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:35.986854 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:35.991066 (+ 4212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:35.991775 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:35.991788 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:35.991795 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:35.991961 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:35.992394 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:35.992402 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:35.992408 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:35.993018 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:35.993289 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:35.993750 (+ 461us) server_negotiation.cc:299] Negotiation successful
0926 03:34:35.994085 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":504,"thread_start_us":318,"threads_started":1}
I20260926 03:34:35.997658 13945 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35187 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:34:35.998993 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:35.998520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35187 (local address 127.0.0.1:51934)
0926 03:34:35.998706 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:35.998851 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 03:34:35.999303 13897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35187: connect: Connection refused (error 111)
I20260926 03:34:36.001250 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:36.000898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35187 (local address 127.0.0.1:51940)
0926 03:34:36.001056 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:36.001141 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 03:34:36.001632 13945 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35187: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35187: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260926 03:34:36.009043 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:35187
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45279,127.0.97.61:44607,127.0.97.60:35187
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:34:36.051046 13945 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35187 } attempt: 1
I20260926 03:34:36.052620 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:36.051963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35187 (local address 127.0.0.1:51942)
0926 03:34:36.052232 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:36.052384 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 03:34:36.053894 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:36.053419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35187 (local address 127.0.0.1:51958)
0926 03:34:36.053591 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:36.053706 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 03:34:36.054137 13945 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35187: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35187: connect: Connection refused (error 111)
I20260926 03:34:36.071476 13871 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44607 } attempt: 5
Sep 26 03:34:36 dist-test-slave-dt7p krb5kdc[13791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 03:34:36.086977 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:36.072502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46274 (local address 127.0.97.61:44607)
0926 03:34:36.072738 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:36.072743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:36.072765 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:34:36.073014 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:36.073019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:36.073095 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:36.073212 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:36.074309 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:36.075125 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:36.076828 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:36.077330 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:36.077490 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:36.081274 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:36.081280 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:36.081285 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:36.081322 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:36.084158 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:36.084939 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:36.084944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:36.084947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:36.085075 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:36.085477 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:36.085481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:36.085483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:36.085777 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:36.085947 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:36.086365 (+ 418us) server_negotiation.cc:299] Negotiation successful
0926 03:34:36.086600 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260926 03:34:36.086977 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:36.072385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44607 (local address 127.0.0.1:46274)
0926 03:34:36.072658 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:36.072683 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 03:34:36.072852 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:34:36.073281 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:34:36.073287 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:34:36.073871 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:34:36.074138 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:36.074152 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:36.075287 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:36.075292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:36.076681 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:36.076697 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:36.077487 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:36.077493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:36.077714 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:36.078434 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:34:36.078459 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:34:36.081070 (+ 2611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:34:36.084319 (+ 3249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:36.084325 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:36.084329 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:36.084796 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:36.085190 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:36.085196 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:36.085200 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:36.085353 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:36.085903 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:34:36.085908 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:34:36.086071 (+ 163us) client_negotiation.cc:769] Sending connection context
0926 03:34:36.086240 (+ 169us) client_negotiation.cc:240] Negotiation successful
0926 03:34:36.086571 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 03:34:36.088406 13871 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35187 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:34:36.089802 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:36.089273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35187 (local address 127.0.0.1:51968)
0926 03:34:36.089523 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:36.089635 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260926 03:34:36.090869 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:36.090500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35187 (local address 127.0.0.1:51972)
0926 03:34:36.090675 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:36.090764 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 03:34:36.091173 13871 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35187: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35187: connect: Connection refused (error 111)
I20260926 03:34:36.122413 13945 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35187 } attempt: 2
I20260926 03:34:36.123965 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:36.123327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35187 (local address 127.0.0.1:51988)
0926 03:34:36.123590 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:36.123718 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260926 03:34:36.125095 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:36.124703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35187 (local address 127.0.0.1:52002)
0926 03:34:36.124862 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:36.124945 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 03:34:36.125319 13945 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35187: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35187: connect: Connection refused (error 111)
I20260926 03:34:36.142552 13871 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35187 } attempt: 1
I20260926 03:34:36.144048 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:36.143485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35187 (local address 127.0.0.1:52012)
0926 03:34:36.143720 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:36.143846 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260926 03:34:36.145170 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:36.144814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35187 (local address 127.0.0.1:52026)
0926 03:34:36.144979 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:36.145062 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 03:34:36.145418 13871 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35187: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35187: connect: Connection refused (error 111)
I20260926 03:34:36.213657 13871 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35187 } attempt: 2
I20260926 03:34:36.215380 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:36.214561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35187 (local address 127.0.0.1:52038)
0926 03:34:36.214919 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:36.215098 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260926 03:34:36.216612 13945 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35187 } attempt: 3
I20260926 03:34:36.216768 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:36.216221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35187 (local address 127.0.0.1:52042)
0926 03:34:36.216416 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:36.216549 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 03:34:36.217054 13871 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35187: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35187: connect: Connection refused (error 111)
I20260926 03:34:36.217867 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:36.217344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35187 (local address 127.0.0.1:52046)
0926 03:34:36.217568 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:36.217671 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 03:34:36.219012 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:36.218534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35187 (local address 127.0.0.1:52054)
0926 03:34:36.218785 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:36.218876 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
W20260926 03:34:36.219341 13945 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35187: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35187: connect: Connection refused (error 111)
I20260926 03:34:36.323333 13871 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35187 } attempt: 3
I20260926 03:34:36.324860 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:36.324250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35187 (local address 127.0.0.1:52068)
0926 03:34:36.324502 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:36.324632 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260926 03:34:36.326102 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:36.325659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35187 (local address 127.0.0.1:52076)
0926 03:34:36.325904 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:36.325985 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260926 03:34:36.326323 13871 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35187: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35187: connect: Connection refused (error 111)
I20260926 03:34:36.362620 13945 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35187 } attempt: 4
I20260926 03:34:36.364078 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:36.363528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35187 (local address 127.0.0.1:52080)
0926 03:34:36.363798 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:36.363901 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 03:34:36.365298 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:36.364943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35187 (local address 127.0.0.1:52096)
0926 03:34:36.365102 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:36.365185 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 03:34:36.365537 13945 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35187: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35187: connect: Connection refused (error 111)
W20260926 03:34:36.402264 13953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:34:36.402730 13953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:34:36.402900 13953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:34:36.411219 13953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:34:36.411423 13953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:34:36.411551 13953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:34:36.411664 13953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:34:36.411747 13953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:34:36.424860 13953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45279,127.0.97.61:44607,127.0.97.60:35187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:35187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:34:36.426692 13953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:34:36.428328 13953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:34:36.442408 13958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:36.442384 13959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:36.443005 13961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:34:36.443499 13953 server_base.cc:1034] running on GCE node
I20260926 03:34:36.444479 13953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:34:36.446277 13953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:34:36.447503 13953 hybrid_clock.cc:648] HybridClock initialized: now 1790393676447461 us; error 29 us; skew 500 ppm
I20260926 03:34:36.447917 13953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:34:36 dist-test-slave-dt7p krb5kdc[13791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:34:36.451538 13953 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260926 03:34:36.453440 13953 webserver.cc:460] Webserver started at http://127.0.97.60:33975/ using document root <none> and password file <none>
I20260926 03:34:36.454270 13953 fs_manager.cc:359] Metadata directory not provided
I20260926 03:34:36.454407 13953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:34:36.454774 13953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:34:36.458315 13953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-2/data/instance:
uuid: "1cabb74d93544ff48e25f0c5168e4ed3"
format_stamp: "Formatted at 2026-09-26 03:34:36 on dist-test-slave-dt7p"
server_key: "262c8111f5ddd981aabc3d2552f02a7a"
server_key_iv: "c9212ac679b101c022ac7c035b1c0476"
server_key_version: "encryptionkey@0"
I20260926 03:34:36.459189 13953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-2/wal/instance:
uuid: "1cabb74d93544ff48e25f0c5168e4ed3"
format_stamp: "Formatted at 2026-09-26 03:34:36 on dist-test-slave-dt7p"
server_key: "262c8111f5ddd981aabc3d2552f02a7a"
server_key_iv: "c9212ac679b101c022ac7c035b1c0476"
server_key_version: "encryptionkey@0"
I20260926 03:34:36.464910 13953 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 03:34:36.468711 13968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:36.470238 13953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:34:36.470423 13953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-2/wal
uuid: "1cabb74d93544ff48e25f0c5168e4ed3"
format_stamp: "Formatted at 2026-09-26 03:34:36 on dist-test-slave-dt7p"
server_key: "262c8111f5ddd981aabc3d2552f02a7a"
server_key_iv: "c9212ac679b101c022ac7c035b1c0476"
server_key_version: "encryptionkey@0"
I20260926 03:34:36.470628 13953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:34:36.484495 13953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:34:36.489271 13953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:34:36.489603 13953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:34:36.503602 13871 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35187 } attempt: 4
I20260926 03:34:36.505200 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:36.504517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35187 (local address 127.0.0.1:52100)
0926 03:34:36.504827 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:36.504980 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260926 03:34:36.506798 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:36.506244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35187 (local address 127.0.0.1:52106)
0926 03:34:36.506466 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:36.506585 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 03:34:36.507124 13871 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35187: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35187: connect: Connection refused (error 111)
I20260926 03:34:36.510790 13953 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:35187
I20260926 03:34:36.510809 14019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:35187 every 8 connection(s)
I20260926 03:34:36.512459 13953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-2/data/info.pb
I20260926 03:34:36.516333 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 13953
I20260926 03:34:36.516553 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790393463219689-388-0/minicluster-data/master-2/wal/instance
I20260926 03:34:36.517350 388 external_mini_cluster.cc:1428] Setting key 0c06ab3bdff7f3ab8096170f78da0050
I20260926 03:34:36.517372 14020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:34:36.528636 14020 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45279 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:34:36 dist-test-slave-dt7p krb5kdc[13791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 26 03:34:36 dist-test-slave-dt7p krb5kdc[13791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:34:36.546419 14023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:36.519514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52122 (local address 127.0.97.60:35187)
0926 03:34:36.520418 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:36.520431 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:36.520482 (+ 51us) server_negotiation.cc:407] Connection header received
0926 03:34:36.522767 (+ 2285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:36.522795 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:36.523295 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:36.523713 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:36.525862 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:36.527587 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:36.529641 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:36.530576 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:36.530748 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:36.536710 (+ 5962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:36.536721 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:36.536737 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:36.536802 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:36.540644 (+ 3842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:36.541662 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:36.541669 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:36.541676 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:36.541875 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:36.542357 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:36.542362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:36.542364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:36.543248 (+ 884us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:36.543511 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:36.544061 (+ 550us) server_negotiation.cc:299] Negotiation successful
0926 03:34:36.544603 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":430,"threads_started":1}
I20260926 03:34:36.550776 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:36.532282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45279 (local address 127.0.0.1:41562)
0926 03:34:36.533229 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:36.533300 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0926 03:34:36.533497 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:34:36.534597 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:34:36.534604 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:34:36.535561 (+ 957us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:34:36.535989 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:36.536003 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:36.537534 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:36.537540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:36.538990 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:36.539002 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:36.540010 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:36.540014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:36.540134 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:36.540854 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:34:36.540891 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:34:36.543815 (+ 2924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:34:36.547810 (+ 3995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:36.547819 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:36.547825 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:36.548221 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:36.548700 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:36.548711 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:36.548713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:36.548845 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:36.549518 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:34:36.549526 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:34:36.549703 (+ 177us) client_negotiation.cc:769] Sending connection context
0926 03:34:36.549950 (+ 247us) client_negotiation.cc:240] Negotiation successful
0926 03:34:36.550271 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":425,"threads_started":1}
I20260926 03:34:36.551373 14025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:36.532414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41562 (local address 127.0.97.62:45279)
0926 03:34:36.534055 (+ 1641us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:36.534065 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:36.534089 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:34:36.534168 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:36.534176 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:36.534313 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:36.534467 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:36.536149 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:36.537332 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:36.539148 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:36.539815 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:36.539979 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:36.544035 (+ 4056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:36.544043 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:36.544048 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:36.544094 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:36.547671 (+ 3577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:36.548356 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:36.548363 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:36.548366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:36.548517 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:36.548973 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:36.548978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:36.548982 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:36.549360 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:36.549548 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:36.550681 (+ 1133us) server_negotiation.cc:299] Negotiation successful
0926 03:34:36.550929 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1489,"thread_start_us":1304,"threads_started":1}
I20260926 03:34:36.552963 14020 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44607 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:34:36 dist-test-slave-dt7p krb5kdc[13791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 26 03:34:36 dist-test-slave-dt7p krb5kdc[13791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 03:34:36.569064 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:36.553929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44607 (local address 127.0.0.1:46282)
0926 03:34:36.554102 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:36.554147 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 03:34:36.554233 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:34:36.554573 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:34:36.554578 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:34:36.555077 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:34:36.555278 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:36.555289 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:36.556324 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:36.556328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:36.557590 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:36.557608 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:36.558377 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:36.558385 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:36.558624 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:36.559176 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:34:36.559203 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:34:36.562298 (+ 3095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:34:36.566227 (+ 3929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:36.566232 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:36.566236 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:36.566777 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:36.567166 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:36.567171 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:36.567175 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:36.567404 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:36.568031 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:34:36.568035 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:34:36.568207 (+ 172us) client_negotiation.cc:769] Sending connection context
0926 03:34:36.568381 (+ 174us) client_negotiation.cc:240] Negotiation successful
0926 03:34:36.568694 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 03:34:36.569214 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:36.550127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41578 (local address 127.0.97.62:45279)
0926 03:34:36.550830 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:36.550837 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:36.550866 (+ 29us) server_negotiation.cc:407] Connection header received
0926 03:34:36.550923 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:36.550928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:36.551009 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:36.551135 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:36.553158 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:36.554409 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:36.556100 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:36.556719 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:36.556820 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:36.562243 (+ 5423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:36.562248 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:36.562252 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:36.562286 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:36.565986 (+ 3700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:36.566861 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:36.566867 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:36.566870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:36.567041 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:36.567510 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:36.567516 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:36.567519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:36.567917 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:36.568190 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:36.568486 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 03:34:36.568751 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":304,"threads_started":1}
I20260926 03:34:36.569555 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:36.553998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46282 (local address 127.0.97.61:44607)
0926 03:34:36.554213 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:36.554219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:36.554240 (+ 21us) server_negotiation.cc:407] Connection header received
0926 03:34:36.554329 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:36.554335 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:36.554433 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:36.554542 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:36.555428 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:36.556173 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:36.557750 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:36.558216 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:36.558321 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:36.562492 (+ 4171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:36.562499 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:36.562503 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:36.562547 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:36.566089 (+ 3542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:36.566882 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:36.566887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:36.566890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:36.567055 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:36.567510 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:36.567516 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:36.567519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:36.567902 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:36.568066 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:36.568492 (+ 426us) server_negotiation.cc:299] Negotiation successful
0926 03:34:36.568746 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 03:34:36.570503 14020 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35187 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:34:36 dist-test-slave-dt7p krb5kdc[13791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 26 03:34:36 dist-test-slave-dt7p krb5kdc[13791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 03:34:36.585136 14023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:36.571432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52128 (local address 127.0.97.60:35187)
0926 03:34:36.571671 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:36.571679 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:36.571845 (+ 166us) server_negotiation.cc:407] Connection header received
0926 03:34:36.572002 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:36.572006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:36.572081 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:36.572184 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:36.572915 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:36.573850 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:36.575435 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:36.576140 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:36.576258 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:36.579510 (+ 3252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:36.579517 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:36.579522 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:36.579560 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:36.582427 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:36.583029 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:36.583036 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:36.583040 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:36.583149 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:36.583557 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:36.583565 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:36.583568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:36.583872 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:36.584025 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:36.584356 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 03:34:36.584695 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":126}
I20260926 03:34:36.585209 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:36.571505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35187 (local address 127.0.0.1:52128)
0926 03:34:36.571771 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:36.571789 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:34:36.571890 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:34:36.572210 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:34:36.572213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:34:36.572636 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:34:36.572774 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:36.572782 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:36.573969 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:36.573974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:36.575299 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:36.575311 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:36.576301 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:36.576308 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:36.576502 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:36.577088 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:34:36.577111 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:34:36.579323 (+ 2212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:34:36.582556 (+ 3233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:36.582563 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:36.582567 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:36.582916 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:36.583261 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:36.583268 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:36.583272 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:36.583446 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:36.583984 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:34:36.583989 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:34:36.584134 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 03:34:36.584270 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 03:34:36.584848 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":230}
I20260926 03:34:36.587657 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:36.571488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46298 (local address 127.0.97.61:44607)
0926 03:34:36.571633 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:36.571639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:36.571656 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:34:36.571793 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:36.571799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:36.571879 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:36.571972 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:36.573510 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:36.574572 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:36.576254 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:36.576959 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:36.577085 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:36.581845 (+ 4760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:36.581850 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:36.581853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:36.581883 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:36.584611 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:36.585582 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:36.585587 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:36.585591 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:36.585746 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:36.586162 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:36.586165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:36.586167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:36.586513 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:36.586791 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:36.587078 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 03:34:36.587369 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 03:34:36.592051 14020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cabb74d93544ff48e25f0c5168e4ed3: Bootstrap starting.
I20260926 03:34:36.593068 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:34:36.596882 14020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cabb74d93544ff48e25f0c5168e4ed3: Neither blocks nor log segments found. Creating new log.
I20260926 03:34:36.598505 14020 log.cc:824] T 00000000000000000000000000000000 P 1cabb74d93544ff48e25f0c5168e4ed3: Log is configured to *not* fsync() on all Append() calls
I20260926 03:34:36.603789 14020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cabb74d93544ff48e25f0c5168e4ed3: No bootstrap required, opened a new log
I20260926 03:34:36.616395 14020 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cabb74d93544ff48e25f0c5168e4ed3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0676e10001d94eeaa8fcbce6c06ebaed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45279 } } peers { permanent_uuid: "164d9567e1db4a32a2946e2d5d76dbaf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44607 } } peers { permanent_uuid: "1cabb74d93544ff48e25f0c5168e4ed3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35187 } }
I20260926 03:34:36.617219 14020 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cabb74d93544ff48e25f0c5168e4ed3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:34:36.617419 14020 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cabb74d93544ff48e25f0c5168e4ed3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cabb74d93544ff48e25f0c5168e4ed3, State: Initialized, Role: FOLLOWER
I20260926 03:34:36.618355 14020 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cabb74d93544ff48e25f0c5168e4ed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0676e10001d94eeaa8fcbce6c06ebaed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45279 } } peers { permanent_uuid: "164d9567e1db4a32a2946e2d5d76dbaf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44607 } } peers { permanent_uuid: "1cabb74d93544ff48e25f0c5168e4ed3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35187 } }
I20260926 03:34:36.621830 14035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cabb74d93544ff48e25f0c5168e4ed3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0676e10001d94eeaa8fcbce6c06ebaed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45279 } } peers { permanent_uuid: "164d9567e1db4a32a2946e2d5d76dbaf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44607 } } peers { permanent_uuid: "1cabb74d93544ff48e25f0c5168e4ed3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35187 } } }
I20260926 03:34:36.623237 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:36.600118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41594 (local address 127.0.97.62:45279)
0926 03:34:36.600393 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:36.600401 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:36.602192 (+ 1791us) server_negotiation.cc:407] Connection header received
0926 03:34:36.602444 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:36.602451 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:36.602556 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:36.602677 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:36.605728 (+ 3051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:36.606902 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:36.609125 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:36.609755 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:36.609875 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:36.614001 (+ 4126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:36.614008 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:36.614013 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:36.614525 (+ 512us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:36.618503 (+ 3978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:36.620471 (+ 1968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:36.620480 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:36.620487 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:36.620705 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:36.621264 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:36.621271 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:36.621278 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:36.621687 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:36.621959 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:36.622595 (+ 636us) server_negotiation.cc:299] Negotiation successful
0926 03:34:36.622892 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 03:34:36.623512 14020 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cabb74d93544ff48e25f0c5168e4ed3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:34:36.624202 14035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cabb74d93544ff48e25f0c5168e4ed3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:34:36.625554 14023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:36.602761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52134 (local address 127.0.97.60:35187)
0926 03:34:36.604100 (+ 1339us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:36.604110 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:36.605254 (+ 1144us) server_negotiation.cc:407] Connection header received
0926 03:34:36.605495 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:36.605501 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:36.605583 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:36.605645 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:36.608403 (+ 2758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:36.609647 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:36.612257 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:36.613045 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:36.613129 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:36.617331 (+ 4202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:36.617338 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:36.617342 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:36.617381 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:36.621202 (+ 3821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:36.622699 (+ 1497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:36.622706 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:36.622711 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:36.622915 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:36.623356 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:36.623361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:36.623365 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:36.623757 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:36.624031 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:36.624988 (+ 957us) server_negotiation.cc:299] Negotiation successful
0926 03:34:36.625169 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1141}
I20260926 03:34:36.627650 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:36.599701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46312 (local address 127.0.97.61:44607)
0926 03:34:36.599916 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:36.599926 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:36.604938 (+ 5012us) server_negotiation.cc:407] Connection header received
0926 03:34:36.605427 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:36.605436 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:36.605526 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:36.605598 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:36.608404 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:36.609613 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:36.612257 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:36.612995 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:36.613105 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:36.617339 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:36.617345 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:36.617349 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:36.617390 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:36.621718 (+ 4328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:36.623104 (+ 1386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:36.623113 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:36.623118 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:36.623296 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:36.625474 (+ 2178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:36.625483 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:36.625489 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:36.625978 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:36.626248 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:36.627206 (+ 958us) server_negotiation.cc:299] Negotiation successful
0926 03:34:36.627403 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 03:34:36.631922 14035 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cabb74d93544ff48e25f0c5168e4ed3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:34:36.632302 14035 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cabb74d93544ff48e25f0c5168e4ed3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0676e10001d94eeaa8fcbce6c06ebaed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45279 } } peers { permanent_uuid: "164d9567e1db4a32a2946e2d5d76dbaf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44607 } } peers { permanent_uuid: "1cabb74d93544ff48e25f0c5168e4ed3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35187 } }
I20260926 03:34:36.634531 14035 leader_election.cc:288] T 00000000000000000000000000000000 P 1cabb74d93544ff48e25f0c5168e4ed3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0676e10001d94eeaa8fcbce6c06ebaed (127.0.97.62:45279), 164d9567e1db4a32a2946e2d5d76dbaf (127.0.97.61:44607)
W20260926 03:34:36.641317 14046 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1cabb74d93544ff48e25f0c5168e4ed3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:34:36.641445 14046 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:34:36.638334 13953 master_runner.cc:186] Error getting master registration for 127.0.97.62:45279: OK, instance_id { permanent_uuid: "0676e10001d94eeaa8fcbce6c06ebaed" instance_seqno: 1790393675346382 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 03:34:36.650020 13953 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 03:34:36.664783 13945 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35187 } attempt: 5
Sep 26 03:34:36 dist-test-slave-dt7p krb5kdc[13791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 03:34:36.683835 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:36.665519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35187 (local address 127.0.0.1:52148)
0926 03:34:36.665745 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:36.665766 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:34:36.665902 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:34:36.666669 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:34:36.666674 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:34:36.667290 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:34:36.667537 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:36.667550 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:36.669092 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:36.669100 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:36.670063 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:36.670075 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:36.671074 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:36.671080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:36.671307 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:36.672085 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:34:36.672109 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:34:36.675261 (+ 3152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:34:36.680827 (+ 5566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:36.680832 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:36.680837 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:36.681423 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:36.681867 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:36.681874 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:36.681877 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:36.682055 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:36.682703 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:34:36.682708 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:34:36.682885 (+ 177us) client_negotiation.cc:769] Sending connection context
0926 03:34:36.683042 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 03:34:36.683399 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 03:34:36.683827 14023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:36.666142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52148 (local address 127.0.97.60:35187)
0926 03:34:36.666342 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:36.666347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:36.666366 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:34:36.666427 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:36.666432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:36.666536 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:36.666632 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:36.667723 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:36.668980 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:36.670212 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:36.670933 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:36.671053 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:36.675438 (+ 4385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:36.675444 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:36.675449 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:36.675494 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:36.680688 (+ 5194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:36.681558 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:36.681566 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:36.681569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:36.681747 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:36.682165 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:36.682170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:36.682174 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:36.682575 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:36.682737 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:36.683147 (+ 410us) server_negotiation.cc:299] Negotiation successful
0926 03:34:36.683416 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 03:34:36.691437 13945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 164d9567e1db4a32a2946e2d5d76dbaf: Bootstrap starting.
I20260926 03:34:36.696386 13945 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 164d9567e1db4a32a2946e2d5d76dbaf: Neither blocks nor log segments found. Creating new log.
I20260926 03:34:36.697994 13945 log.cc:824] T 00000000000000000000000000000000 P 164d9567e1db4a32a2946e2d5d76dbaf: Log is configured to *not* fsync() on all Append() calls
I20260926 03:34:36.701776 13945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 164d9567e1db4a32a2946e2d5d76dbaf: No bootstrap required, opened a new log
I20260926 03:34:36.713622 13945 raft_consensus.cc:348] T 00000000000000000000000000000000 P 164d9567e1db4a32a2946e2d5d76dbaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0676e10001d94eeaa8fcbce6c06ebaed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45279 } } peers { permanent_uuid: "164d9567e1db4a32a2946e2d5d76dbaf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44607 } } peers { permanent_uuid: "1cabb74d93544ff48e25f0c5168e4ed3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35187 } }
I20260926 03:34:36.714090 13945 raft_consensus.cc:374] T 00000000000000000000000000000000 P 164d9567e1db4a32a2946e2d5d76dbaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:34:36.714218 13945 raft_consensus.cc:729] T 00000000000000000000000000000000 P 164d9567e1db4a32a2946e2d5d76dbaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 164d9567e1db4a32a2946e2d5d76dbaf, State: Initialized, Role: FOLLOWER
I20260926 03:34:36.714967 13945 consensus_queue.cc:260] T 00000000000000000000000000000000 P 164d9567e1db4a32a2946e2d5d76dbaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0676e10001d94eeaa8fcbce6c06ebaed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45279 } } peers { permanent_uuid: "164d9567e1db4a32a2946e2d5d76dbaf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44607 } } peers { permanent_uuid: "1cabb74d93544ff48e25f0c5168e4ed3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35187 } }
I20260926 03:34:36.716105 14049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 164d9567e1db4a32a2946e2d5d76dbaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0676e10001d94eeaa8fcbce6c06ebaed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45279 } } peers { permanent_uuid: "164d9567e1db4a32a2946e2d5d76dbaf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44607 } } peers { permanent_uuid: "1cabb74d93544ff48e25f0c5168e4ed3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35187 } } }
I20260926 03:34:36.716506 14049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 164d9567e1db4a32a2946e2d5d76dbaf [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:34:36.717571 13945 sys_catalog.cc:564] T 00000000000000000000000000000000 P 164d9567e1db4a32a2946e2d5d76dbaf [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:34:36.717634 13920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1cabb74d93544ff48e25f0c5168e4ed3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "164d9567e1db4a32a2946e2d5d76dbaf" is_pre_election: true
I20260926 03:34:36.718333 13920 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 164d9567e1db4a32a2946e2d5d76dbaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cabb74d93544ff48e25f0c5168e4ed3 in term 0.
I20260926 03:34:36.720695 13970 leader_election.cc:302] T 00000000000000000000000000000000 P 1cabb74d93544ff48e25f0c5168e4ed3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 164d9567e1db4a32a2946e2d5d76dbaf, 1cabb74d93544ff48e25f0c5168e4ed3; no voters:
I20260926 03:34:36.721339 14035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cabb74d93544ff48e25f0c5168e4ed3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:34:36.721521 14035 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cabb74d93544ff48e25f0c5168e4ed3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:34:36.721634 14035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cabb74d93544ff48e25f0c5168e4ed3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:34:36.725870 14035 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cabb74d93544ff48e25f0c5168e4ed3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0676e10001d94eeaa8fcbce6c06ebaed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45279 } } peers { permanent_uuid: "164d9567e1db4a32a2946e2d5d76dbaf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44607 } } peers { permanent_uuid: "1cabb74d93544ff48e25f0c5168e4ed3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35187 } }
I20260926 03:34:36.727193 13920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1cabb74d93544ff48e25f0c5168e4ed3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "164d9567e1db4a32a2946e2d5d76dbaf"
I20260926 03:34:36.727437 13920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 164d9567e1db4a32a2946e2d5d76dbaf [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:34:36.731205 13920 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 164d9567e1db4a32a2946e2d5d76dbaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cabb74d93544ff48e25f0c5168e4ed3 in term 1.
I20260926 03:34:36.731829 13970 leader_election.cc:302] T 00000000000000000000000000000000 P 1cabb74d93544ff48e25f0c5168e4ed3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 164d9567e1db4a32a2946e2d5d76dbaf, 1cabb74d93544ff48e25f0c5168e4ed3; no voters:
I20260926 03:34:36.732591 14035 leader_election.cc:288] T 00000000000000000000000000000000 P 1cabb74d93544ff48e25f0c5168e4ed3 [CANDIDATE]: Term 1 election: Requested vote from peers 0676e10001d94eeaa8fcbce6c06ebaed (127.0.97.62:45279), 164d9567e1db4a32a2946e2d5d76dbaf (127.0.97.61:44607)
I20260926 03:34:36.733081 14059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cabb74d93544ff48e25f0c5168e4ed3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:34:36.733341 14059 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cabb74d93544ff48e25f0c5168e4ed3 [term 1 LEADER]: Becoming Leader. State: Replica: 1cabb74d93544ff48e25f0c5168e4ed3, State: Running, Role: LEADER
I20260926 03:34:36.734072 14059 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cabb74d93544ff48e25f0c5168e4ed3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0676e10001d94eeaa8fcbce6c06ebaed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45279 } } peers { permanent_uuid: "164d9567e1db4a32a2946e2d5d76dbaf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44607 } } peers { permanent_uuid: "1cabb74d93544ff48e25f0c5168e4ed3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35187 } }
I20260926 03:34:36.741396 14035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cabb74d93544ff48e25f0c5168e4ed3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cabb74d93544ff48e25f0c5168e4ed3. Latest consensus state: current_term: 1 leader_uuid: "1cabb74d93544ff48e25f0c5168e4ed3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0676e10001d94eeaa8fcbce6c06ebaed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45279 } } peers { permanent_uuid: "164d9567e1db4a32a2946e2d5d76dbaf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44607 } } peers { permanent_uuid: "1cabb74d93544ff48e25f0c5168e4ed3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35187 } } }
I20260926 03:34:36.741765 14035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cabb74d93544ff48e25f0c5168e4ed3 [sys.catalog]: This master's current role is: LEADER
W20260926 03:34:36.742003 13974 tablet.cc:2369] T 00000000000000000000000000000000 P 1cabb74d93544ff48e25f0c5168e4ed3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:34:36.751194 14063 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 03:34:36.752115 14062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:34:36.755144 14062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:34:36.753954 13878 master_runner.cc:186] Error getting master registration for 127.0.97.62:45279: OK, instance_id { permanent_uuid: "0676e10001d94eeaa8fcbce6c06ebaed" instance_seqno: 1790393675346382 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 03:34:36.765890 13878 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 03:34:36.766040 13920 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 164d9567e1db4a32a2946e2d5d76dbaf [term 1 FOLLOWER]: Refusing update from remote peer 1cabb74d93544ff48e25f0c5168e4ed3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:34:36.767045 14035 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1cabb74d93544ff48e25f0c5168e4ed3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "164d9567e1db4a32a2946e2d5d76dbaf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:34:36.777482 14049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 164d9567e1db4a32a2946e2d5d76dbaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cabb74d93544ff48e25f0c5168e4ed3. Latest consensus state: current_term: 1 leader_uuid: "1cabb74d93544ff48e25f0c5168e4ed3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0676e10001d94eeaa8fcbce6c06ebaed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45279 } } peers { permanent_uuid: "164d9567e1db4a32a2946e2d5d76dbaf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44607 } } peers { permanent_uuid: "1cabb74d93544ff48e25f0c5168e4ed3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35187 } } }
I20260926 03:34:36.777832 14049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 164d9567e1db4a32a2946e2d5d76dbaf [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:34:36.781116 14059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cabb74d93544ff48e25f0c5168e4ed3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1cabb74d93544ff48e25f0c5168e4ed3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0676e10001d94eeaa8fcbce6c06ebaed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45279 } } peers { permanent_uuid: "164d9567e1db4a32a2946e2d5d76dbaf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44607 } } peers { permanent_uuid: "1cabb74d93544ff48e25f0c5168e4ed3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35187 } } }
I20260926 03:34:36.781599 14059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cabb74d93544ff48e25f0c5168e4ed3 [sys.catalog]: This master's current role is: LEADER
I20260926 03:34:36.781996 14065 mvcc.cc:205] Tried to move back new op lower bound from 7333452500021469184 to 7333452499909320704. Current Snapshot: MvccSnapshot[applied={T|T < 7333452500021469184}]
I20260926 03:34:36.784365 14062 catalog_manager.cc:1347] Generated new cluster ID: dc381db58bf94190ab321642b7a17b1a
I20260926 03:34:36.784482 14062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:34:36.784402 13871 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35187 } attempt: 5
I20260926 03:34:36.785171 14049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 164d9567e1db4a32a2946e2d5d76dbaf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1cabb74d93544ff48e25f0c5168e4ed3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0676e10001d94eeaa8fcbce6c06ebaed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45279 } } peers { permanent_uuid: "164d9567e1db4a32a2946e2d5d76dbaf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44607 } } peers { permanent_uuid: "1cabb74d93544ff48e25f0c5168e4ed3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35187 } } }
I20260926 03:34:36.785419 14049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 164d9567e1db4a32a2946e2d5d76dbaf [sys.catalog]: This master's current role is: FOLLOWER
Sep 26 03:34:36 dist-test-slave-dt7p krb5kdc[13791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 03:34:36.815610 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:36.789034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35187 (local address 127.0.0.1:52156)
0926 03:34:36.789273 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:36.789306 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 03:34:36.789470 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:34:36.789950 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:34:36.789956 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:34:36.790530 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:34:36.790804 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:36.790817 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:36.792384 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:36.792389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:36.793843 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:36.793858 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:36.797109 (+ 3251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:36.797115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:36.797321 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:36.798146 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:34:36.798171 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:34:36.801650 (+ 3479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:34:36.809081 (+ 7431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:36.809087 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:36.809092 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:36.810088 (+ 996us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:36.810682 (+ 594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:36.810688 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:36.810692 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:36.810881 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:36.812127 (+ 1246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:34:36.812132 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:34:36.812366 (+ 234us) client_negotiation.cc:769] Sending connection context
0926 03:34:36.814794 (+ 2428us) client_negotiation.cc:240] Negotiation successful
0926 03:34:36.815149 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 03:34:36.816809 14023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:36.789059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52156 (local address 127.0.97.60:35187)
0926 03:34:36.789257 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:36.789265 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:36.789596 (+ 331us) server_negotiation.cc:407] Connection header received
0926 03:34:36.789679 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:36.789685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:36.789776 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:36.789880 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:36.790979 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:36.792222 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:36.796119 (+ 3897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:36.796925 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:36.797071 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:36.801881 (+ 4810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:36.801888 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:36.801894 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:36.801936 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:36.808894 (+ 6958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:36.810261 (+ 1367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:36.810267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:36.810271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:36.810507 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:36.811084 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:36.811091 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:36.811126 (+ 35us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:36.811962 (+ 836us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:36.814345 (+ 2383us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:36.815123 (+ 778us) server_negotiation.cc:299] Negotiation successful
0926 03:34:36.815815 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 03:34:36.823580 13871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0676e10001d94eeaa8fcbce6c06ebaed: Bootstrap starting.
I20260926 03:34:36.827716 14062 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:34:36.828763 13871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0676e10001d94eeaa8fcbce6c06ebaed: Neither blocks nor log segments found. Creating new log.
I20260926 03:34:36.830657 13871 log.cc:824] T 00000000000000000000000000000000 P 0676e10001d94eeaa8fcbce6c06ebaed: Log is configured to *not* fsync() on all Append() calls
I20260926 03:34:36.832041 14062 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:34:36.843058 13871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0676e10001d94eeaa8fcbce6c06ebaed: No bootstrap required, opened a new log
I20260926 03:34:36.855091 13871 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0676e10001d94eeaa8fcbce6c06ebaed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0676e10001d94eeaa8fcbce6c06ebaed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45279 } } peers { permanent_uuid: "164d9567e1db4a32a2946e2d5d76dbaf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44607 } } peers { permanent_uuid: "1cabb74d93544ff48e25f0c5168e4ed3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35187 } }
I20260926 03:34:36.855571 13871 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0676e10001d94eeaa8fcbce6c06ebaed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:34:36.855715 13871 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0676e10001d94eeaa8fcbce6c06ebaed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0676e10001d94eeaa8fcbce6c06ebaed, State: Initialized, Role: FOLLOWER
I20260926 03:34:36.856477 13871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0676e10001d94eeaa8fcbce6c06ebaed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0676e10001d94eeaa8fcbce6c06ebaed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45279 } } peers { permanent_uuid: "164d9567e1db4a32a2946e2d5d76dbaf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44607 } } peers { permanent_uuid: "1cabb74d93544ff48e25f0c5168e4ed3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35187 } }
I20260926 03:34:36.858724 14062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cabb74d93544ff48e25f0c5168e4ed3: Generated new TSK 0
I20260926 03:34:36.859879 14062 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:34:36.863958 14070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0676e10001d94eeaa8fcbce6c06ebaed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0676e10001d94eeaa8fcbce6c06ebaed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45279 } } peers { permanent_uuid: "164d9567e1db4a32a2946e2d5d76dbaf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44607 } } peers { permanent_uuid: "1cabb74d93544ff48e25f0c5168e4ed3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35187 } } }
I20260926 03:34:36.864409 14070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0676e10001d94eeaa8fcbce6c06ebaed [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:34:36.866612 13871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0676e10001d94eeaa8fcbce6c06ebaed [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:34:36.866982 13845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1cabb74d93544ff48e25f0c5168e4ed3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0676e10001d94eeaa8fcbce6c06ebaed"
I20260926 03:34:36.867434 13844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0676e10001d94eeaa8fcbce6c06ebaed [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:34:36.866674 13846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1cabb74d93544ff48e25f0c5168e4ed3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0676e10001d94eeaa8fcbce6c06ebaed" is_pre_election: true
I20260926 03:34:36.870929 13844 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0676e10001d94eeaa8fcbce6c06ebaed [term 1 FOLLOWER]: Refusing update from remote peer 1cabb74d93544ff48e25f0c5168e4ed3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:34:36.871677 14035 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1cabb74d93544ff48e25f0c5168e4ed3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0676e10001d94eeaa8fcbce6c06ebaed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:34:36.917747 14070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0676e10001d94eeaa8fcbce6c06ebaed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cabb74d93544ff48e25f0c5168e4ed3. Latest consensus state: current_term: 1 leader_uuid: "1cabb74d93544ff48e25f0c5168e4ed3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0676e10001d94eeaa8fcbce6c06ebaed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45279 } } peers { permanent_uuid: "164d9567e1db4a32a2946e2d5d76dbaf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44607 } } peers { permanent_uuid: "1cabb74d93544ff48e25f0c5168e4ed3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35187 } } }
I20260926 03:34:36.918260 14070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0676e10001d94eeaa8fcbce6c06ebaed [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:34:36.918682 14081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0676e10001d94eeaa8fcbce6c06ebaed [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1cabb74d93544ff48e25f0c5168e4ed3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0676e10001d94eeaa8fcbce6c06ebaed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45279 } } peers { permanent_uuid: "164d9567e1db4a32a2946e2d5d76dbaf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44607 } } peers { permanent_uuid: "1cabb74d93544ff48e25f0c5168e4ed3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35187 } } }
I20260926 03:34:36.919209 14081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0676e10001d94eeaa8fcbce6c06ebaed [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:34:36.927364 14088 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 03:34:37.287930 14035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cabb74d93544ff48e25f0c5168e4ed3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1cabb74d93544ff48e25f0c5168e4ed3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0676e10001d94eeaa8fcbce6c06ebaed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45279 } } peers { permanent_uuid: "164d9567e1db4a32a2946e2d5d76dbaf" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44607 } } peers { permanent_uuid: "1cabb74d93544ff48e25f0c5168e4ed3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35187 } } }
I20260926 03:34:37.288303 14035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cabb74d93544ff48e25f0c5168e4ed3 [sys.catalog]: This master's current role is: LEADER
I20260926 03:34:37.756781 14063 catalog_manager.cc:1259] Loaded cluster ID: dc381db58bf94190ab321642b7a17b1a
I20260926 03:34:37.757030 14063 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 164d9567e1db4a32a2946e2d5d76dbaf: loading cluster ID for follower catalog manager: success
I20260926 03:34:37.760151 14063 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 164d9567e1db4a32a2946e2d5d76dbaf: acquiring CA information for follower catalog manager: success
I20260926 03:34:37.762212 14063 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 164d9567e1db4a32a2946e2d5d76dbaf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 03:34:37.933444 14088 catalog_manager.cc:1259] Loaded cluster ID: dc381db58bf94190ab321642b7a17b1a
I20260926 03:34:37.933651 14088 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0676e10001d94eeaa8fcbce6c06ebaed: loading cluster ID for follower catalog manager: success
I20260926 03:34:37.937747 14088 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0676e10001d94eeaa8fcbce6c06ebaed: acquiring CA information for follower catalog manager: success
I20260926 03:34:37.940639 14088 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0676e10001d94eeaa8fcbce6c06ebaed: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 03:34:37.991312 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 13804
I20260926 03:34:37.993582 13866 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:34:38.103181 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 13804
I20260926 03:34:38.131547 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 13878
I20260926 03:34:38.133064 13940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:34:38.261567 14097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:38.260785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45279 (local address 127.0.0.1:41604)
0926 03:34:38.261214 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:38.261357 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":206,"threads_started":1}
W20260926 03:34:38.262146 13969 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45279: connect: Connection refused (error 111)
I20260926 03:34:38.265457 14097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:38.264888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45279 (local address 127.0.0.1:41620)
0926 03:34:38.265173 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:38.265284 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260926 03:34:38.265440 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 13878
W20260926 03:34:38.266256 13969 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1cabb74d93544ff48e25f0c5168e4ed3 -> Peer 0676e10001d94eeaa8fcbce6c06ebaed (127.0.97.62:45279): Couldn't send request to peer 0676e10001d94eeaa8fcbce6c06ebaed. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45279: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 03:34:38.289757 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 13953
I20260926 03:34:38.291074 14015 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:34:38.416908 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 13953
2026-09-26T03:34:38Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4996 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:34:38 dist-test-slave-dt7p krb5kdc[14102](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:34:38 dist-test-slave-dt7p krb5kdc[14102](info): set up 2 sockets
Sep 26 03:34:38 dist-test-slave-dt7p krb5kdc[14102](info): commencing operation
krb5kdc: starting...
W20260926 03:34:39.533241 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.008s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:34:39 dist-test-slave-dt7p krb5kdc[14102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:34:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:34:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:34:39.805490 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41995
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41995
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:34:40.217698 14115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:34:40.218063 14115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:34:40.218256 14115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:34:40.226498 14115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:34:40.226643 14115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:34:40.226794 14115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:34:40.226907 14115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:34:40.226992 14115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:34:40.238196 14115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:34:40.239913 14115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:34:40.241542 14115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:34:40.254998 14120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:40.255089 14121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:40.255113 14123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:34:40.256333 14115 server_base.cc:1034] running on GCE node
I20260926 03:34:40.257390 14115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:34:40.259130 14115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:34:40.260350 14115 hybrid_clock.cc:648] HybridClock initialized: now 1790393680260290 us; error 43 us; skew 500 ppm
I20260926 03:34:40.260804 14115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:34:40 dist-test-slave-dt7p krb5kdc[14102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:34:40.264398 14115 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:34:40.266149 14115 webserver.cc:460] Webserver started at http://127.0.97.62:35645/ using document root <none> and password file <none>
I20260926 03:34:40.266927 14115 fs_manager.cc:359] Metadata directory not provided
I20260926 03:34:40.267028 14115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:34:40.267325 14115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:34:40.270660 14115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "4b771dd5715342d1b01b94fa692df5b4"
format_stamp: "Formatted at 2026-09-26 03:34:40 on dist-test-slave-dt7p"
server_key: "306eb8423e27d4aa505d23349d0d1f91"
server_key_iv: "d820a8bbb7d0eb5dbb8b6d4a3a2136f0"
server_key_version: "encryptionkey@0"
I20260926 03:34:40.271430 14115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "4b771dd5715342d1b01b94fa692df5b4"
format_stamp: "Formatted at 2026-09-26 03:34:40 on dist-test-slave-dt7p"
server_key: "306eb8423e27d4aa505d23349d0d1f91"
server_key_iv: "d820a8bbb7d0eb5dbb8b6d4a3a2136f0"
server_key_version: "encryptionkey@0"
I20260926 03:34:40.277125 14115 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 03:34:40.280800 14130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:40.282302 14115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:34:40.282506 14115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "4b771dd5715342d1b01b94fa692df5b4"
format_stamp: "Formatted at 2026-09-26 03:34:40 on dist-test-slave-dt7p"
server_key: "306eb8423e27d4aa505d23349d0d1f91"
server_key_iv: "d820a8bbb7d0eb5dbb8b6d4a3a2136f0"
server_key_version: "encryptionkey@0"
I20260926 03:34:40.282708 14115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:34:40.303263 14115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:34:40.307762 14115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:34:40.308080 14115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:34:40.329417 14115 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41995
I20260926 03:34:40.329442 14181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41995 every 8 connection(s)
I20260926 03:34:40.331188 14115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:34:40.333866 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 14115
I20260926 03:34:40.334129 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:34:40.335083 388 external_mini_cluster.cc:1428] Setting key 1a449268140dfe807a77091eb72735bb
I20260926 03:34:40.337002 14182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 03:34:40 dist-test-slave-dt7p krb5kdc[14102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:34:40.353775 14182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b771dd5715342d1b01b94fa692df5b4: Bootstrap starting.
I20260926 03:34:40.357650 14182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b771dd5715342d1b01b94fa692df5b4: Neither blocks nor log segments found. Creating new log.
I20260926 03:34:40.358881 14182 log.cc:824] T 00000000000000000000000000000000 P 4b771dd5715342d1b01b94fa692df5b4: Log is configured to *not* fsync() on all Append() calls
I20260926 03:34:40.361864 14185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:40.337306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34422 (local address 127.0.97.62:41995)
0926 03:34:40.338168 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:40.338181 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:40.338228 (+ 47us) server_negotiation.cc:407] Connection header received
0926 03:34:40.340045 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:40.340068 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:40.340515 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:40.340973 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:40.342990 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:40.345091 (+ 2101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:40.347057 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:40.347804 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:40.347951 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:40.353586 (+ 5635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:40.353595 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:40.353615 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:40.353666 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:40.357068 (+ 3402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:40.357910 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:40.357918 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:40.357925 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:40.358125 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:40.358520 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:40.358526 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:40.358529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:40.359199 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:40.359539 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:40.360036 (+ 497us) server_negotiation.cc:299] Negotiation successful
0926 03:34:40.360753 (+ 717us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":392,"threads_started":1}
I20260926 03:34:40.362352 14182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b771dd5715342d1b01b94fa692df5b4: No bootstrap required, opened a new log
I20260926 03:34:40.366284 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:34:40.372506 14182 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b771dd5715342d1b01b94fa692df5b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b771dd5715342d1b01b94fa692df5b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41995 } }
I20260926 03:34:40.373037 14182 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b771dd5715342d1b01b94fa692df5b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:34:40.373121 14182 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b771dd5715342d1b01b94fa692df5b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b771dd5715342d1b01b94fa692df5b4, State: Initialized, Role: FOLLOWER
I20260926 03:34:40.373773 14182 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b771dd5715342d1b01b94fa692df5b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b771dd5715342d1b01b94fa692df5b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41995 } }
I20260926 03:34:40.374001 14182 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b771dd5715342d1b01b94fa692df5b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:34:40.374111 14182 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b771dd5715342d1b01b94fa692df5b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:34:40.374264 14182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b771dd5715342d1b01b94fa692df5b4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:34:40.376708 14182 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b771dd5715342d1b01b94fa692df5b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b771dd5715342d1b01b94fa692df5b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41995 } }
I20260926 03:34:40.377260 14182 leader_election.cc:302] T 00000000000000000000000000000000 P 4b771dd5715342d1b01b94fa692df5b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b771dd5715342d1b01b94fa692df5b4; no voters:
I20260926 03:34:40.377856 14182 leader_election.cc:288] T 00000000000000000000000000000000 P 4b771dd5715342d1b01b94fa692df5b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:34:40.378032 14187 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b771dd5715342d1b01b94fa692df5b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:34:40.378746 14187 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b771dd5715342d1b01b94fa692df5b4 [term 1 LEADER]: Becoming Leader. State: Replica: 4b771dd5715342d1b01b94fa692df5b4, State: Running, Role: LEADER
I20260926 03:34:40.379324 14187 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b771dd5715342d1b01b94fa692df5b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b771dd5715342d1b01b94fa692df5b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41995 } }
I20260926 03:34:40.380319 14182 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b771dd5715342d1b01b94fa692df5b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:34:40.382704 14188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b771dd5715342d1b01b94fa692df5b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b771dd5715342d1b01b94fa692df5b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b771dd5715342d1b01b94fa692df5b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41995 } } }
I20260926 03:34:40.383111 14188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b771dd5715342d1b01b94fa692df5b4 [sys.catalog]: This master's current role is: LEADER
I20260926 03:34:40.383524 14189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b771dd5715342d1b01b94fa692df5b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b771dd5715342d1b01b94fa692df5b4. Latest consensus state: current_term: 1 leader_uuid: "4b771dd5715342d1b01b94fa692df5b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b771dd5715342d1b01b94fa692df5b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41995 } } }
I20260926 03:34:40.383832 14189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b771dd5715342d1b01b94fa692df5b4 [sys.catalog]: This master's current role is: LEADER
I20260926 03:34:40.385437 14194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:34:40.391847 14194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:34:40.401453 14194 catalog_manager.cc:1347] Generated new cluster ID: 3d70517735434bcdb7ed37ec2c1834e1
I20260926 03:34:40.401587 14194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:34:40.417984 14194 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:34:40.419523 14194 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:34:40.436064 14194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b771dd5715342d1b01b94fa692df5b4: Generated new TSK 0
I20260926 03:34:40.437570 14194 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:34:40.481472 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 14115
I20260926 03:34:40.482918 14177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:34:40.589138 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 14115
2026-09-26T03:34:40Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2166 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:34:40 dist-test-slave-dt7p krb5kdc[14210](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:34:40 dist-test-slave-dt7p krb5kdc[14210](info): set up 2 sockets
Sep 26 03:34:40 dist-test-slave-dt7p krb5kdc[14210](info): commencing operation
krb5kdc: starting...
W20260926 03:34:41.697474 388 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.007s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:34:41 dist-test-slave-dt7p krb5kdc[14210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:34:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:34:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:34:41.974402 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46679
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46679
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:34:42.361863 14223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:34:42.362236 14223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:34:42.362344 14223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:34:42.370363 14223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:34:42.370496 14223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:34:42.370548 14223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:34:42.370612 14223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:34:42.370659 14223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:34:42.382017 14223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:34:42.383682 14223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:34:42.385334 14223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:34:42.398415 14228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:42.398388 14231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:42.398643 14229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:34:42.399302 14223 server_base.cc:1034] running on GCE node
I20260926 03:34:42.400166 14223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:34:42.401921 14223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:34:42.403172 14223 hybrid_clock.cc:648] HybridClock initialized: now 1790393682403093 us; error 81 us; skew 500 ppm
I20260926 03:34:42.403558 14223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:34:42 dist-test-slave-dt7p krb5kdc[14210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:34:42.407222 14223 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:34:42.408968 14223 webserver.cc:460] Webserver started at http://127.0.97.62:35667/ using document root <none> and password file <none>
I20260926 03:34:42.409797 14223 fs_manager.cc:359] Metadata directory not provided
I20260926 03:34:42.409955 14223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:34:42.410305 14223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:34:42.413915 14223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "57587beacbcd425cb35e2e05c483e514"
format_stamp: "Formatted at 2026-09-26 03:34:42 on dist-test-slave-dt7p"
server_key: "024b68f856ef40629c1bb4e9181d454e"
server_key_iv: "a59f70e151481a7f3ee2bb49d7763389"
server_key_version: "encryptionkey@0"
I20260926 03:34:42.414759 14223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "57587beacbcd425cb35e2e05c483e514"
format_stamp: "Formatted at 2026-09-26 03:34:42 on dist-test-slave-dt7p"
server_key: "024b68f856ef40629c1bb4e9181d454e"
server_key_iv: "a59f70e151481a7f3ee2bb49d7763389"
server_key_version: "encryptionkey@0"
I20260926 03:34:42.420740 14223 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 03:34:42.424782 14238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:42.426349 14223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:34:42.426546 14223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "57587beacbcd425cb35e2e05c483e514"
format_stamp: "Formatted at 2026-09-26 03:34:42 on dist-test-slave-dt7p"
server_key: "024b68f856ef40629c1bb4e9181d454e"
server_key_iv: "a59f70e151481a7f3ee2bb49d7763389"
server_key_version: "encryptionkey@0"
I20260926 03:34:42.426787 14223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:34:42.442938 14223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:34:42.448686 14223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:34:42.449003 14223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:34:42.469842 14223 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46679
I20260926 03:34:42.469861 14289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46679 every 8 connection(s)
I20260926 03:34:42.471500 14223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:34:42.472880 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 14223
I20260926 03:34:42.473124 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:34:42.473852 388 external_mini_cluster.cc:1428] Setting key 286142d27cc56a48b6319ec332376f64
I20260926 03:34:42.476655 14290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 03:34:42 dist-test-slave-dt7p krb5kdc[14210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:34:42.490921 14290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514: Bootstrap starting.
I20260926 03:34:42.495693 14290 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514: Neither blocks nor log segments found. Creating new log.
I20260926 03:34:42.497170 14290 log.cc:824] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514: Log is configured to *not* fsync() on all Append() calls
I20260926 03:34:42.500633 14290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514: No bootstrap required, opened a new log
I20260926 03:34:42.500756 14293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:42.476085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38622 (local address 127.0.97.62:46679)
0926 03:34:42.476972 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:42.476996 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:42.477052 (+ 56us) server_negotiation.cc:407] Connection header received
0926 03:34:42.479531 (+ 2479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:42.479563 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:42.480118 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:42.480592 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:42.482043 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:42.484196 (+ 2153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:42.485706 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:42.486813 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:42.487013 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:42.491717 (+ 4704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:42.491728 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:42.491750 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:42.491822 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:42.496025 (+ 4203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:42.496828 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:42.496837 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:42.496847 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:42.497068 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:42.497562 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:42.497567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:42.497570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:42.498301 (+ 731us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:42.498652 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:42.499149 (+ 497us) server_negotiation.cc:299] Negotiation successful
0926 03:34:42.499604 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":393,"threads_started":1}
I20260926 03:34:42.509604 14290 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57587beacbcd425cb35e2e05c483e514" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46679 } }
I20260926 03:34:42.510072 14290 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:34:42.510196 14290 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57587beacbcd425cb35e2e05c483e514, State: Initialized, Role: FOLLOWER
I20260926 03:34:42.510895 14290 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57587beacbcd425cb35e2e05c483e514" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46679 } }
I20260926 03:34:42.511179 14290 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:34:42.511332 14290 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:34:42.511518 14290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:34:42.514266 14290 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57587beacbcd425cb35e2e05c483e514" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46679 } }
I20260926 03:34:42.514842 14290 leader_election.cc:302] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57587beacbcd425cb35e2e05c483e514; no voters:
I20260926 03:34:42.515511 14290 leader_election.cc:288] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:34:42.515669 14295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:34:42.517051 14295 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [term 1 LEADER]: Becoming Leader. State: Replica: 57587beacbcd425cb35e2e05c483e514, State: Running, Role: LEADER
I20260926 03:34:42.517587 14295 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57587beacbcd425cb35e2e05c483e514" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46679 } }
I20260926 03:34:42.518005 14290 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:34:42.520941 14297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57587beacbcd425cb35e2e05c483e514. Latest consensus state: current_term: 1 leader_uuid: "57587beacbcd425cb35e2e05c483e514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57587beacbcd425cb35e2e05c483e514" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46679 } } }
I20260926 03:34:42.520923 14296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57587beacbcd425cb35e2e05c483e514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57587beacbcd425cb35e2e05c483e514" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46679 } } }
I20260926 03:34:42.521339 14297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [sys.catalog]: This master's current role is: LEADER
I20260926 03:34:42.521392 14296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [sys.catalog]: This master's current role is: LEADER
I20260926 03:34:42.523546 14302 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:34:42.529461 14302 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:34:42.539093 14302 catalog_manager.cc:1347] Generated new cluster ID: 04964e8c9e844d89b322d8c905759b1d
I20260926 03:34:42.539237 14302 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:34:42.554323 14302 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:34:42.556407 14302 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:34:42.565356 14302 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514: Generated new TSK 0
I20260926 03:34:42.566532 14302 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:34:42.637976 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46679
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:34:43.028700 14316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:34:43.029048 14316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:34:43.029217 14316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:34:43.037349 14316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:34:43.037492 14316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:34:43.037747 14316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:34:43.048616 14316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:34:43.050307 14316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:34:43.052032 14316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:34:43.066856 14322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:43.066896 14321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:43.066792 14324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:34:43.067796 14316 server_base.cc:1034] running on GCE node
I20260926 03:34:43.068468 14316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:34:43.069533 14316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:34:43.070740 14316 hybrid_clock.cc:648] HybridClock initialized: now 1790393683070687 us; error 39 us; skew 500 ppm
I20260926 03:34:43.071040 14316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:34:43 dist-test-slave-dt7p krb5kdc[14210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:34:43.074546 14316 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:34:43.076077 14316 webserver.cc:460] Webserver started at http://127.0.97.1:35465/ using document root <none> and password file <none>
I20260926 03:34:43.076956 14316 fs_manager.cc:359] Metadata directory not provided
I20260926 03:34:43.077065 14316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:34:43.077358 14316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:34:43.080729 14316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/data/instance:
uuid: "3511b88ddb2643189dceaea58e35ce9d"
format_stamp: "Formatted at 2026-09-26 03:34:43 on dist-test-slave-dt7p"
server_key: "cc8b2b244a878c26f943c89be916d847"
server_key_iv: "1301223c8b27074a4b02588290d98e64"
server_key_version: "encryptionkey@0"
I20260926 03:34:43.081548 14316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3511b88ddb2643189dceaea58e35ce9d"
format_stamp: "Formatted at 2026-09-26 03:34:43 on dist-test-slave-dt7p"
server_key: "cc8b2b244a878c26f943c89be916d847"
server_key_iv: "1301223c8b27074a4b02588290d98e64"
server_key_version: "encryptionkey@0"
I20260926 03:34:43.087177 14316 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:34:43.091087 14331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:43.092612 14316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:34:43.092757 14316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/wal
uuid: "3511b88ddb2643189dceaea58e35ce9d"
format_stamp: "Formatted at 2026-09-26 03:34:43 on dist-test-slave-dt7p"
server_key: "cc8b2b244a878c26f943c89be916d847"
server_key_iv: "1301223c8b27074a4b02588290d98e64"
server_key_version: "encryptionkey@0"
I20260926 03:34:43.092947 14316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:34:43.112923 14316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:34:43.117523 14316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:34:43.117797 14316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:34:43.118990 14316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:34:43.120575 14316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:34:43.120682 14316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260926 03:34:43.120781 14316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:34:43.120827 14316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:43.161178 14316 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41037
I20260926 03:34:43.161123 14443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41037 every 8 connection(s)
I20260926 03:34:43.163501 14316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:34:43.172835 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 14316
I20260926 03:34:43.173123 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:34:43.174150 388 external_mini_cluster.cc:1428] Setting key e6a1010e60ada60cd369e2b1c33cf26d
Sep 26 03:34:43 dist-test-slave-dt7p krb5kdc[14210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:34:43.197512 14446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:43.167549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48699 (local address 127.0.97.62:46679)
0926 03:34:43.168417 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:43.168422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:43.169513 (+ 1091us) server_negotiation.cc:407] Connection header received
0926 03:34:43.172377 (+ 2864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:43.172383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:43.172500 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:43.172652 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:43.175273 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:43.176398 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:43.179530 (+ 3132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:43.180289 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:43.180439 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:43.186063 (+ 5624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:43.186073 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:43.186082 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:43.186135 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:43.190660 (+ 4525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:43.191790 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:43.191796 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:43.191800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:43.191973 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:43.192453 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:43.192459 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:43.192464 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:43.192917 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:43.193085 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:43.196794 (+ 3709us) server_negotiation.cc:299] Negotiation successful
0926 03:34:43.197082 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":561,"threads_started":1}
I20260926 03:34:43.198179 14447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:43.168157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.1:48699)
0926 03:34:43.169154 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:43.169207 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 03:34:43.171961 (+ 2754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:34:43.173207 (+ 1246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:34:43.173223 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:34:43.174212 (+ 989us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:34:43.175026 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:43.175052 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:43.176662 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:43.176695 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:43.179310 (+ 2615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:43.179349 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:43.180848 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:43.180858 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:43.181225 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:43.182214 (+ 989us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:34:43.182272 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:34:43.185785 (+ 3513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:34:43.190833 (+ 5048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:43.190848 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:43.190875 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:43.191650 (+ 775us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:43.192091 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:43.192097 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:43.192101 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:43.192325 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:43.193163 (+ 838us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:34:43.193172 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:34:43.193764 (+ 592us) client_negotiation.cc:769] Sending connection context
0926 03:34:43.194140 (+ 376us) client_negotiation.cc:240] Negotiation successful
0926 03:34:43.194884 (+ 744us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":436,"threads_started":1}
I20260926 03:34:43.200309 14444 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46679
I20260926 03:34:43.200853 14444 heartbeater.cc:464] Registering TS with master...
I20260926 03:34:43.201993 14444 heartbeater.cc:511] Master 127.0.97.62:46679 requested a full tablet report, sending...
I20260926 03:34:43.205261 14255 ts_manager.cc:194] Registered new tserver with Master: 3511b88ddb2643189dceaea58e35ce9d (127.0.97.1:41037)
I20260926 03:34:43.207614 14255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48699
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:34:43.248369 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46679
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:34:43.630096 14450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:34:43.630476 14450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:34:43.630659 14450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:34:43.638621 14450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:34:43.638749 14450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:34:43.638960 14450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:34:43.649839 14450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:34:43.651639 14450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:34:43.653469 14450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:34:43.668799 14456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:43.668818 14455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:43.669004 14458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:34:43.670617 14450 server_base.cc:1034] running on GCE node
I20260926 03:34:43.671370 14450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:34:43.672549 14450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:34:43.673872 14450 hybrid_clock.cc:648] HybridClock initialized: now 1790393683673821 us; error 37 us; skew 500 ppm
I20260926 03:34:43.674255 14450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:34:43 dist-test-slave-dt7p krb5kdc[14210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:34:43.678157 14450 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:34:43.680035 14450 webserver.cc:460] Webserver started at http://127.0.97.2:40145/ using document root <none> and password file <none>
I20260926 03:34:43.681056 14450 fs_manager.cc:359] Metadata directory not provided
I20260926 03:34:43.681237 14450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:34:43.681646 14450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:34:43.685513 14450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/data/instance:
uuid: "1cff45dfdc4c48bc9decf5f844982927"
format_stamp: "Formatted at 2026-09-26 03:34:43 on dist-test-slave-dt7p"
server_key: "8a1a4c57ef90077cdce1ecdd3e73f827"
server_key_iv: "5afae4554131e9a0c80d726fa118248e"
server_key_version: "encryptionkey@0"
I20260926 03:34:43.686551 14450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1cff45dfdc4c48bc9decf5f844982927"
format_stamp: "Formatted at 2026-09-26 03:34:43 on dist-test-slave-dt7p"
server_key: "8a1a4c57ef90077cdce1ecdd3e73f827"
server_key_iv: "5afae4554131e9a0c80d726fa118248e"
server_key_version: "encryptionkey@0"
I20260926 03:34:43.693378 14450 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 03:34:43.697582 14465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:43.699294 14450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260926 03:34:43.699501 14450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/wal
uuid: "1cff45dfdc4c48bc9decf5f844982927"
format_stamp: "Formatted at 2026-09-26 03:34:43 on dist-test-slave-dt7p"
server_key: "8a1a4c57ef90077cdce1ecdd3e73f827"
server_key_iv: "5afae4554131e9a0c80d726fa118248e"
server_key_version: "encryptionkey@0"
I20260926 03:34:43.699756 14450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:34:43.748658 14450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:34:43.753502 14450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:34:43.753844 14450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:34:43.754989 14450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:34:43.756723 14450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:34:43.756836 14450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:43.756994 14450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:34:43.757068 14450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:43.795367 14450 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35639
I20260926 03:34:43.795470 14577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35639 every 8 connection(s)
I20260926 03:34:43.797158 14450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:34:43.807567 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 14450
I20260926 03:34:43.807801 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:34:43.808357 388 external_mini_cluster.cc:1428] Setting key a030667dc5ba2d56f6cbc6f71459d20d
Sep 26 03:34:43 dist-test-slave-dt7p krb5kdc[14210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:34:43.822568 14580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:43.800298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53863 (local address 127.0.97.62:46679)
0926 03:34:43.800866 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:43.800871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:43.801936 (+ 1065us) server_negotiation.cc:407] Connection header received
0926 03:34:43.804131 (+ 2195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:43.804138 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:43.804249 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:43.804389 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:43.806256 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:43.807062 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:43.809043 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:43.809672 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:43.809826 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:43.814696 (+ 4870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:43.814706 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:43.814714 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:43.814769 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:43.818803 (+ 4034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:43.819789 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:43.819797 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:43.819803 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:43.819972 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:43.820440 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:43.820448 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:43.820454 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:43.820906 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:43.821078 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:43.821877 (+ 799us) server_negotiation.cc:299] Negotiation successful
0926 03:34:43.822137 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":312,"threads_started":1}
I20260926 03:34:43.823865 14581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:43.800934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.2:53863)
0926 03:34:43.801739 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:43.801787 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 03:34:43.803805 (+ 2018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:34:43.804714 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:34:43.804725 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:34:43.805371 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:34:43.806047 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:43.806065 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:43.807239 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:43.807244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:43.808884 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:43.808897 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:43.809922 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:43.809931 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:43.810194 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:43.811164 (+ 970us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:34:43.811199 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:34:43.814389 (+ 3190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:34:43.818957 (+ 4568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:43.818966 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:43.818983 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:43.819648 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:43.820088 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:43.820093 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:43.820097 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:43.820315 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:43.821042 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:34:43.821049 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:34:43.821447 (+ 398us) client_negotiation.cc:769] Sending connection context
0926 03:34:43.821776 (+ 329us) client_negotiation.cc:240] Negotiation successful
0926 03:34:43.822418 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":342,"threads_started":1}
I20260926 03:34:43.825456 14578 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46679
I20260926 03:34:43.825850 14578 heartbeater.cc:464] Registering TS with master...
I20260926 03:34:43.826843 14578 heartbeater.cc:511] Master 127.0.97.62:46679 requested a full tablet report, sending...
I20260926 03:34:43.828684 14255 ts_manager.cc:194] Registered new tserver with Master: 1cff45dfdc4c48bc9decf5f844982927 (127.0.97.2:35639)
I20260926 03:34:43.829680 14255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53863
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:34:43.872313 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46679
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:34:44.211530 14444 heartbeater.cc:503] Master 127.0.97.62:46679 was elected leader, sending a full tablet report...
W20260926 03:34:44.252063 14584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:34:44.252408 14584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:34:44.252640 14584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:34:44.260439 14584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:34:44.260599 14584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:34:44.260804 14584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:34:44.271572 14584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:34:44.273325 14584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:34:44.274909 14584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:34:44.289602 14589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:44.289566 14592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:44.289659 14590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:34:44.291231 14584 server_base.cc:1034] running on GCE node
I20260926 03:34:44.291893 14584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:34:44.293099 14584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:34:44.294304 14584 hybrid_clock.cc:648] HybridClock initialized: now 1790393684294259 us; error 34 us; skew 500 ppm
I20260926 03:34:44.294664 14584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:34:44 dist-test-slave-dt7p krb5kdc[14210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:34:44.298234 14584 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:34:44.299855 14584 webserver.cc:460] Webserver started at http://127.0.97.3:38903/ using document root <none> and password file <none>
I20260926 03:34:44.300714 14584 fs_manager.cc:359] Metadata directory not provided
I20260926 03:34:44.300864 14584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:34:44.301198 14584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:34:44.304668 14584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/data/instance:
uuid: "cb9ebb92eee54c939139cef3b151f9c9"
format_stamp: "Formatted at 2026-09-26 03:34:44 on dist-test-slave-dt7p"
server_key: "6148b146126b8f37a04fe7a627f95e41"
server_key_iv: "bae1a42b5a61105223a88eeeb1c898fa"
server_key_version: "encryptionkey@0"
I20260926 03:34:44.305526 14584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/wal/instance:
uuid: "cb9ebb92eee54c939139cef3b151f9c9"
format_stamp: "Formatted at 2026-09-26 03:34:44 on dist-test-slave-dt7p"
server_key: "6148b146126b8f37a04fe7a627f95e41"
server_key_iv: "bae1a42b5a61105223a88eeeb1c898fa"
server_key_version: "encryptionkey@0"
I20260926 03:34:44.311127 14584 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:34:44.314841 14599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:44.316277 14584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:34:44.316457 14584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/wal
uuid: "cb9ebb92eee54c939139cef3b151f9c9"
format_stamp: "Formatted at 2026-09-26 03:34:44 on dist-test-slave-dt7p"
server_key: "6148b146126b8f37a04fe7a627f95e41"
server_key_iv: "bae1a42b5a61105223a88eeeb1c898fa"
server_key_version: "encryptionkey@0"
I20260926 03:34:44.316720 14584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:34:44.335415 14584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:34:44.340297 14584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:34:44.340677 14584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:34:44.341837 14584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:34:44.343505 14584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:34:44.343612 14584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:44.343782 14584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:34:44.343863 14584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:44.381870 14584 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45531
I20260926 03:34:44.381906 14711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45531 every 8 connection(s)
I20260926 03:34:44.383431 14584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:34:44.389478 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 14584
I20260926 03:34:44.389700 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:34:44.390331 388 external_mini_cluster.cc:1428] Setting key 4b629b6c3841a51d8a65cd8c0dd3746b
Sep 26 03:34:44 dist-test-slave-dt7p krb5kdc[14210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:34:44.406607 14714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:44.386556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55085 (local address 127.0.97.62:46679)
0926 03:34:44.387177 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:44.387182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:44.388018 (+ 836us) server_negotiation.cc:407] Connection header received
0926 03:34:44.390782 (+ 2764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:44.390792 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:44.390893 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:44.391043 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:44.392857 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:44.393888 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:44.395512 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:44.395948 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:44.396081 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:44.400145 (+ 4064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:44.400150 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:44.400154 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:44.400191 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:44.402887 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:44.403670 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:44.403674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:44.403677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:44.403818 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:44.404378 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:44.404382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:44.404384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:44.404846 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:44.405024 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:44.405917 (+ 893us) server_negotiation.cc:299] Negotiation successful
0926 03:34:44.406182 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":339,"threads_started":1}
I20260926 03:34:44.407829 14715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:44.387012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.3:55085)
0926 03:34:44.387829 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:44.387871 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 03:34:44.390410 (+ 2539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:34:44.391332 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:34:44.391343 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:34:44.391921 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:34:44.392673 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:44.392694 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:44.394042 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:44.394047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:44.395363 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:44.395374 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:44.396195 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:44.396203 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:44.396464 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:44.397390 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:34:44.397469 (+ 79us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:34:44.399948 (+ 2479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:34:44.403015 (+ 3067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:44.403022 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:44.403038 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:44.403554 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:44.403952 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:44.403956 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:44.403960 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:44.404231 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:44.404984 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:34:44.404994 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:34:44.405493 (+ 499us) client_negotiation.cc:769] Sending connection context
0926 03:34:44.405802 (+ 309us) client_negotiation.cc:240] Negotiation successful
0926 03:34:44.406399 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":324,"threads_started":1}
I20260926 03:34:44.409363 14712 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46679
I20260926 03:34:44.409739 14712 heartbeater.cc:464] Registering TS with master...
I20260926 03:34:44.410622 14712 heartbeater.cc:511] Master 127.0.97.62:46679 requested a full tablet report, sending...
I20260926 03:34:44.412220 14255 ts_manager.cc:194] Registered new tserver with Master: cb9ebb92eee54c939139cef3b151f9c9 (127.0.97.3:45531)
I20260926 03:34:44.413180 14255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55085
I20260926 03:34:44.418697 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:34:44.437511 14714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:44.423900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38626 (local address 127.0.97.62:46679)
0926 03:34:44.424103 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:44.424112 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:44.424625 (+ 513us) server_negotiation.cc:407] Connection header received
0926 03:34:44.424881 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:44.424886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:44.424968 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:44.425083 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:44.426500 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:44.427330 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:44.428736 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:44.429152 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:44.429315 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:44.431858 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:44.431865 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:44.431876 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:44.431913 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:44.434554 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:44.435282 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:44.435289 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:44.435293 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:44.435427 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:44.435823 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:44.435829 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:44.435832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:44.436201 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:44.436503 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:44.436880 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 03:34:44.437143 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 03:34:44.441960 14255 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38626:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:34:44.449698 14255 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:34:44.489991 14727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:44.465757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35639 (local address 127.0.0.1:45628)
0926 03:34:44.466872 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:44.466912 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 03:34:44.467155 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:34:44.468437 (+ 1282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:34:44.468449 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:34:44.468490 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:34:44.480616 (+ 12126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:44.480632 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:44.484176 (+ 3544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:44.484181 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:44.486710 (+ 2529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:44.486728 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:44.489005 (+ 2277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:44.489012 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:44.489272 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:44.489306 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 03:34:44.489381 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 03:34:44.489532 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"mutex_wait_us":21,"thread_start_us":1280,"threads_started":1}
I20260926 03:34:44.490824 14728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:44.466144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37402 (local address 127.0.97.3:45531)
0926 03:34:44.478117 (+ 11973us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:44.478131 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:44.478167 (+ 36us) server_negotiation.cc:407] Connection header received
0926 03:34:44.478273 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:44.478281 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:44.478572 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:44.478778 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:34:44.479684 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:44.482155 (+ 2471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:44.486155 (+ 4000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:44.487649 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:44.487774 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:44.488022 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:44.490246 (+ 2224us) server_negotiation.cc:299] Negotiation successful
0926 03:34:44.490456 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11773,"thread_start_us":11595,"threads_started":1}
I20260926 03:34:44.491056 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:44.465558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45628 (local address 127.0.97.2:35639)
0926 03:34:44.466362 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:44.466374 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:44.467016 (+ 642us) server_negotiation.cc:407] Connection header received
0926 03:34:44.467417 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:44.467426 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:44.467730 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:44.467974 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:34:44.480887 (+ 12913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:44.484008 (+ 3121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:44.486916 (+ 2908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:44.488485 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:44.488640 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:44.488862 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:44.490465 (+ 1603us) server_negotiation.cc:299] Negotiation successful
0926 03:34:44.490675 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":411,"threads_started":1}
I20260926 03:34:44.491146 14725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:44.465601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45531 (local address 127.0.0.1:37402)
0926 03:34:44.466874 (+ 1273us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:44.466913 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 03:34:44.467151 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:34:44.478808 (+ 11657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:34:44.478816 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:34:44.478855 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:34:44.479463 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:44.479482 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:44.482332 (+ 2850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:44.482338 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:44.485969 (+ 3631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:44.485982 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:44.488188 (+ 2206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:44.488194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:44.488431 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:44.488484 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 03:34:44.488646 (+ 162us) client_negotiation.cc:240] Negotiation successful
0926 03:34:44.488873 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":925,"thread_start_us":754,"threads_started":1}
I20260926 03:34:44.493327 14729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:44.467485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41037 (local address 127.0.0.1:55710)
0926 03:34:44.477022 (+ 9537us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:44.477048 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 03:34:44.477209 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:34:44.484349 (+ 7140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:34:44.484355 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:34:44.484390 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:34:44.484988 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:44.485010 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:44.487013 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:44.487018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:44.489273 (+ 2255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:44.489285 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:44.492453 (+ 3168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:44.492458 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:44.492647 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:44.492680 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 03:34:44.492757 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 03:34:44.492917 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9329,"mutex_wait_us":25,"thread_start_us":9111,"threads_started":1}
I20260926 03:34:44.494405 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:44.467875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55710 (local address 127.0.97.1:41037)
0926 03:34:44.483267 (+ 15392us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:44.483277 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:44.483309 (+ 32us) server_negotiation.cc:407] Connection header received
0926 03:34:44.483418 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:44.483425 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:44.483674 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:44.483863 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:34:44.485243 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:44.486854 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:44.489455 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:44.490943 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:44.491070 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:44.493781 (+ 2711us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:44.493863 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 03:34:44.494051 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15163,"thread_start_us":16175,"threads_started":1}
I20260926 03:34:44.495791 14513 tablet_service.cc:1467] Processing CreateTablet for tablet 660703b9ca474be282d6118fa64cbc25 (DEFAULT_TABLE table=test-table [id=1c62ae4fcb7942c6a95eb9bc00993f7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:34:44.495790 14647 tablet_service.cc:1467] Processing CreateTablet for tablet 660703b9ca474be282d6118fa64cbc25 (DEFAULT_TABLE table=test-table [id=1c62ae4fcb7942c6a95eb9bc00993f7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:34:44.497308 14513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 660703b9ca474be282d6118fa64cbc25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:34:44.497335 14647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 660703b9ca474be282d6118fa64cbc25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:34:44.498364 14379 tablet_service.cc:1467] Processing CreateTablet for tablet 660703b9ca474be282d6118fa64cbc25 (DEFAULT_TABLE table=test-table [id=1c62ae4fcb7942c6a95eb9bc00993f7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:34:44.499936 14379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 660703b9ca474be282d6118fa64cbc25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:34:44.509965 14731 tablet_bootstrap.cc:492] T 660703b9ca474be282d6118fa64cbc25 P 1cff45dfdc4c48bc9decf5f844982927: Bootstrap starting.
I20260926 03:34:44.513432 14732 tablet_bootstrap.cc:492] T 660703b9ca474be282d6118fa64cbc25 P cb9ebb92eee54c939139cef3b151f9c9: Bootstrap starting.
I20260926 03:34:44.515060 14731 tablet_bootstrap.cc:654] T 660703b9ca474be282d6118fa64cbc25 P 1cff45dfdc4c48bc9decf5f844982927: Neither blocks nor log segments found. Creating new log.
I20260926 03:34:44.515460 14733 tablet_bootstrap.cc:492] T 660703b9ca474be282d6118fa64cbc25 P 3511b88ddb2643189dceaea58e35ce9d: Bootstrap starting.
I20260926 03:34:44.516726 14731 log.cc:824] T 660703b9ca474be282d6118fa64cbc25 P 1cff45dfdc4c48bc9decf5f844982927: Log is configured to *not* fsync() on all Append() calls
I20260926 03:34:44.517181 14732 tablet_bootstrap.cc:654] T 660703b9ca474be282d6118fa64cbc25 P cb9ebb92eee54c939139cef3b151f9c9: Neither blocks nor log segments found. Creating new log.
I20260926 03:34:44.518476 14733 tablet_bootstrap.cc:654] T 660703b9ca474be282d6118fa64cbc25 P 3511b88ddb2643189dceaea58e35ce9d: Neither blocks nor log segments found. Creating new log.
I20260926 03:34:44.518777 14732 log.cc:824] T 660703b9ca474be282d6118fa64cbc25 P cb9ebb92eee54c939139cef3b151f9c9: Log is configured to *not* fsync() on all Append() calls
I20260926 03:34:44.519598 14733 log.cc:824] T 660703b9ca474be282d6118fa64cbc25 P 3511b88ddb2643189dceaea58e35ce9d: Log is configured to *not* fsync() on all Append() calls
I20260926 03:34:44.522608 14732 tablet_bootstrap.cc:492] T 660703b9ca474be282d6118fa64cbc25 P cb9ebb92eee54c939139cef3b151f9c9: No bootstrap required, opened a new log
I20260926 03:34:44.522924 14732 ts_tablet_manager.cc:1397] T 660703b9ca474be282d6118fa64cbc25 P cb9ebb92eee54c939139cef3b151f9c9: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 03:34:44.523306 14733 tablet_bootstrap.cc:492] T 660703b9ca474be282d6118fa64cbc25 P 3511b88ddb2643189dceaea58e35ce9d: No bootstrap required, opened a new log
I20260926 03:34:44.523577 14731 tablet_bootstrap.cc:492] T 660703b9ca474be282d6118fa64cbc25 P 1cff45dfdc4c48bc9decf5f844982927: No bootstrap required, opened a new log
I20260926 03:34:44.523561 14733 ts_tablet_manager.cc:1397] T 660703b9ca474be282d6118fa64cbc25 P 3511b88ddb2643189dceaea58e35ce9d: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 03:34:44.523854 14731 ts_tablet_manager.cc:1397] T 660703b9ca474be282d6118fa64cbc25 P 1cff45dfdc4c48bc9decf5f844982927: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260926 03:34:44.531730 14732 raft_consensus.cc:348] T 660703b9ca474be282d6118fa64cbc25 P cb9ebb92eee54c939139cef3b151f9c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff45dfdc4c48bc9decf5f844982927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35639 } } peers { permanent_uuid: "cb9ebb92eee54c939139cef3b151f9c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45531 } } peers { permanent_uuid: "3511b88ddb2643189dceaea58e35ce9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41037 } }
I20260926 03:34:44.532325 14732 raft_consensus.cc:374] T 660703b9ca474be282d6118fa64cbc25 P cb9ebb92eee54c939139cef3b151f9c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:34:44.532442 14732 raft_consensus.cc:729] T 660703b9ca474be282d6118fa64cbc25 P cb9ebb92eee54c939139cef3b151f9c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb9ebb92eee54c939139cef3b151f9c9, State: Initialized, Role: FOLLOWER
I20260926 03:34:44.533345 14732 consensus_queue.cc:260] T 660703b9ca474be282d6118fa64cbc25 P cb9ebb92eee54c939139cef3b151f9c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff45dfdc4c48bc9decf5f844982927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35639 } } peers { permanent_uuid: "cb9ebb92eee54c939139cef3b151f9c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45531 } } peers { permanent_uuid: "3511b88ddb2643189dceaea58e35ce9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41037 } }
I20260926 03:34:44.534618 14712 heartbeater.cc:503] Master 127.0.97.62:46679 was elected leader, sending a full tablet report...
I20260926 03:34:44.534859 14733 raft_consensus.cc:348] T 660703b9ca474be282d6118fa64cbc25 P 3511b88ddb2643189dceaea58e35ce9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff45dfdc4c48bc9decf5f844982927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35639 } } peers { permanent_uuid: "cb9ebb92eee54c939139cef3b151f9c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45531 } } peers { permanent_uuid: "3511b88ddb2643189dceaea58e35ce9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41037 } }
I20260926 03:34:44.534868 14731 raft_consensus.cc:348] T 660703b9ca474be282d6118fa64cbc25 P 1cff45dfdc4c48bc9decf5f844982927 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff45dfdc4c48bc9decf5f844982927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35639 } } peers { permanent_uuid: "cb9ebb92eee54c939139cef3b151f9c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45531 } } peers { permanent_uuid: "3511b88ddb2643189dceaea58e35ce9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41037 } }
I20260926 03:34:44.535300 14731 raft_consensus.cc:374] T 660703b9ca474be282d6118fa64cbc25 P 1cff45dfdc4c48bc9decf5f844982927 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:34:44.535315 14733 raft_consensus.cc:374] T 660703b9ca474be282d6118fa64cbc25 P 3511b88ddb2643189dceaea58e35ce9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:34:44.535398 14732 ts_tablet_manager.cc:1428] T 660703b9ca474be282d6118fa64cbc25 P cb9ebb92eee54c939139cef3b151f9c9: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260926 03:34:44.535398 14731 raft_consensus.cc:729] T 660703b9ca474be282d6118fa64cbc25 P 1cff45dfdc4c48bc9decf5f844982927 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cff45dfdc4c48bc9decf5f844982927, State: Initialized, Role: FOLLOWER
I20260926 03:34:44.535409 14733 raft_consensus.cc:729] T 660703b9ca474be282d6118fa64cbc25 P 3511b88ddb2643189dceaea58e35ce9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3511b88ddb2643189dceaea58e35ce9d, State: Initialized, Role: FOLLOWER
I20260926 03:34:44.536016 14733 consensus_queue.cc:260] T 660703b9ca474be282d6118fa64cbc25 P 3511b88ddb2643189dceaea58e35ce9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff45dfdc4c48bc9decf5f844982927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35639 } } peers { permanent_uuid: "cb9ebb92eee54c939139cef3b151f9c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45531 } } peers { permanent_uuid: "3511b88ddb2643189dceaea58e35ce9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41037 } }
I20260926 03:34:44.536062 14731 consensus_queue.cc:260] T 660703b9ca474be282d6118fa64cbc25 P 1cff45dfdc4c48bc9decf5f844982927 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff45dfdc4c48bc9decf5f844982927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35639 } } peers { permanent_uuid: "cb9ebb92eee54c939139cef3b151f9c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45531 } } peers { permanent_uuid: "3511b88ddb2643189dceaea58e35ce9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41037 } }
I20260926 03:34:44.538025 14731 ts_tablet_manager.cc:1428] T 660703b9ca474be282d6118fa64cbc25 P 1cff45dfdc4c48bc9decf5f844982927: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260926 03:34:44.538304 14733 ts_tablet_manager.cc:1428] T 660703b9ca474be282d6118fa64cbc25 P 3511b88ddb2643189dceaea58e35ce9d: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260926 03:34:44.539132 14578 heartbeater.cc:503] Master 127.0.97.62:46679 was elected leader, sending a full tablet report...
W20260926 03:34:44.549453 14579 tablet.cc:2369] T 660703b9ca474be282d6118fa64cbc25 P 1cff45dfdc4c48bc9decf5f844982927: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:34:44.635442 14713 tablet.cc:2369] T 660703b9ca474be282d6118fa64cbc25 P cb9ebb92eee54c939139cef3b151f9c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:34:44.667025 14445 tablet.cc:2369] T 660703b9ca474be282d6118fa64cbc25 P 3511b88ddb2643189dceaea58e35ce9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:34:44.718178 14738 raft_consensus.cc:482] T 660703b9ca474be282d6118fa64cbc25 P 3511b88ddb2643189dceaea58e35ce9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:34:44.718605 14738 raft_consensus.cc:504] T 660703b9ca474be282d6118fa64cbc25 P 3511b88ddb2643189dceaea58e35ce9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff45dfdc4c48bc9decf5f844982927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35639 } } peers { permanent_uuid: "cb9ebb92eee54c939139cef3b151f9c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45531 } } peers { permanent_uuid: "3511b88ddb2643189dceaea58e35ce9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41037 } }
I20260926 03:34:44.721210 14738 leader_election.cc:288] T 660703b9ca474be282d6118fa64cbc25 P 3511b88ddb2643189dceaea58e35ce9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cff45dfdc4c48bc9decf5f844982927 (127.0.97.2:35639), cb9ebb92eee54c939139cef3b151f9c9 (127.0.97.3:45531)
I20260926 03:34:44.729619 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:44.722303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49383 (local address 127.0.97.2:35639)
0926 03:34:44.722548 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:44.722555 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:44.722625 (+ 70us) server_negotiation.cc:407] Connection header received
0926 03:34:44.722924 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:44.722933 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:44.723041 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:44.723146 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:34:44.724000 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:44.725160 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:44.727315 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:44.728487 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:44.728662 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:44.728755 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:44.729116 (+ 361us) server_negotiation.cc:299] Negotiation successful
0926 03:34:44.729289 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 03:34:44.729768 14740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:44.721559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35639 (local address 127.0.97.1:49383)
0926 03:34:44.722529 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:44.722553 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:34:44.722746 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:34:44.723313 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:34:44.723318 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:34:44.723356 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:34:44.723772 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:44.723782 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:44.725298 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:44.725303 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:44.727120 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:44.727141 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:44.728740 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:44.728747 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:44.728933 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:44.728974 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 03:34:44.729093 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 03:34:44.729293 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"mutex_wait_us":1,"thread_start_us":846,"threads_started":1}
I20260926 03:34:44.730219 14741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:44.721613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45531 (local address 127.0.97.1:37893)
0926 03:34:44.723561 (+ 1948us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:44.723576 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:34:44.723739 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:34:44.724156 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:34:44.724161 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:34:44.724187 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:34:44.724667 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:44.724679 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:44.726073 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:44.726102 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:44.727794 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:44.727811 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:44.729362 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:44.729372 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:44.729583 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:44.729625 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 03:34:44.729734 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 03:34:44.729873 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1836,"thread_start_us":1427,"threads_started":1}
I20260926 03:34:44.730239 14728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:44.721699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37893 (local address 127.0.97.3:45531)
0926 03:34:44.721956 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:44.721965 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:44.723683 (+ 1718us) server_negotiation.cc:407] Connection header received
0926 03:34:44.723925 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:44.723934 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:44.724020 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:44.724094 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:34:44.724850 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:44.725904 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:44.727982 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:44.729098 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:44.729216 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:44.729322 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:44.729790 (+ 468us) server_negotiation.cc:299] Negotiation successful
0926 03:34:44.729953 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 03:34:44.731359 14667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660703b9ca474be282d6118fa64cbc25" candidate_uuid: "3511b88ddb2643189dceaea58e35ce9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb9ebb92eee54c939139cef3b151f9c9" is_pre_election: true
I20260926 03:34:44.731333 14533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660703b9ca474be282d6118fa64cbc25" candidate_uuid: "3511b88ddb2643189dceaea58e35ce9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cff45dfdc4c48bc9decf5f844982927" is_pre_election: true
I20260926 03:34:44.731978 14667 raft_consensus.cc:2454] T 660703b9ca474be282d6118fa64cbc25 P cb9ebb92eee54c939139cef3b151f9c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3511b88ddb2643189dceaea58e35ce9d in term 0.
I20260926 03:34:44.731978 14533 raft_consensus.cc:2454] T 660703b9ca474be282d6118fa64cbc25 P 1cff45dfdc4c48bc9decf5f844982927 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3511b88ddb2643189dceaea58e35ce9d in term 0.
I20260926 03:34:44.732853 14333 leader_election.cc:302] T 660703b9ca474be282d6118fa64cbc25 P 3511b88ddb2643189dceaea58e35ce9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cff45dfdc4c48bc9decf5f844982927, 3511b88ddb2643189dceaea58e35ce9d; no voters:
I20260926 03:34:44.733469 14738 raft_consensus.cc:2781] T 660703b9ca474be282d6118fa64cbc25 P 3511b88ddb2643189dceaea58e35ce9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:34:44.733642 14738 raft_consensus.cc:482] T 660703b9ca474be282d6118fa64cbc25 P 3511b88ddb2643189dceaea58e35ce9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:34:44.733777 14738 raft_consensus.cc:3037] T 660703b9ca474be282d6118fa64cbc25 P 3511b88ddb2643189dceaea58e35ce9d [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:34:44.736328 14738 raft_consensus.cc:504] T 660703b9ca474be282d6118fa64cbc25 P 3511b88ddb2643189dceaea58e35ce9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff45dfdc4c48bc9decf5f844982927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35639 } } peers { permanent_uuid: "cb9ebb92eee54c939139cef3b151f9c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45531 } } peers { permanent_uuid: "3511b88ddb2643189dceaea58e35ce9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41037 } }
I20260926 03:34:44.737061 14738 leader_election.cc:288] T 660703b9ca474be282d6118fa64cbc25 P 3511b88ddb2643189dceaea58e35ce9d [CANDIDATE]: Term 1 election: Requested vote from peers 1cff45dfdc4c48bc9decf5f844982927 (127.0.97.2:35639), cb9ebb92eee54c939139cef3b151f9c9 (127.0.97.3:45531)
I20260926 03:34:44.737679 14667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660703b9ca474be282d6118fa64cbc25" candidate_uuid: "3511b88ddb2643189dceaea58e35ce9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb9ebb92eee54c939139cef3b151f9c9"
I20260926 03:34:44.737673 14533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660703b9ca474be282d6118fa64cbc25" candidate_uuid: "3511b88ddb2643189dceaea58e35ce9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cff45dfdc4c48bc9decf5f844982927"
I20260926 03:34:44.737944 14533 raft_consensus.cc:3037] T 660703b9ca474be282d6118fa64cbc25 P 1cff45dfdc4c48bc9decf5f844982927 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:34:44.737952 14667 raft_consensus.cc:3037] T 660703b9ca474be282d6118fa64cbc25 P cb9ebb92eee54c939139cef3b151f9c9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:34:44.740443 14667 raft_consensus.cc:2454] T 660703b9ca474be282d6118fa64cbc25 P cb9ebb92eee54c939139cef3b151f9c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3511b88ddb2643189dceaea58e35ce9d in term 1.
I20260926 03:34:44.740442 14533 raft_consensus.cc:2454] T 660703b9ca474be282d6118fa64cbc25 P 1cff45dfdc4c48bc9decf5f844982927 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3511b88ddb2643189dceaea58e35ce9d in term 1.
I20260926 03:34:44.741024 14332 leader_election.cc:302] T 660703b9ca474be282d6118fa64cbc25 P 3511b88ddb2643189dceaea58e35ce9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1cff45dfdc4c48bc9decf5f844982927, 3511b88ddb2643189dceaea58e35ce9d, cb9ebb92eee54c939139cef3b151f9c9; no voters:
I20260926 03:34:44.741407 14738 raft_consensus.cc:2781] T 660703b9ca474be282d6118fa64cbc25 P 3511b88ddb2643189dceaea58e35ce9d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:34:44.741920 14738 raft_consensus.cc:686] T 660703b9ca474be282d6118fa64cbc25 P 3511b88ddb2643189dceaea58e35ce9d [term 1 LEADER]: Becoming Leader. State: Replica: 3511b88ddb2643189dceaea58e35ce9d, State: Running, Role: LEADER
I20260926 03:34:44.742513 14738 consensus_queue.cc:237] T 660703b9ca474be282d6118fa64cbc25 P 3511b88ddb2643189dceaea58e35ce9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff45dfdc4c48bc9decf5f844982927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35639 } } peers { permanent_uuid: "cb9ebb92eee54c939139cef3b151f9c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45531 } } peers { permanent_uuid: "3511b88ddb2643189dceaea58e35ce9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41037 } }
I20260926 03:34:44.748194 14254 catalog_manager.cc:5495] T 660703b9ca474be282d6118fa64cbc25 P 3511b88ddb2643189dceaea58e35ce9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3511b88ddb2643189dceaea58e35ce9d (127.0.97.1). New cstate: current_term: 1 leader_uuid: "3511b88ddb2643189dceaea58e35ce9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff45dfdc4c48bc9decf5f844982927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb9ebb92eee54c939139cef3b151f9c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3511b88ddb2643189dceaea58e35ce9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41037 } health_report { overall_health: HEALTHY } } }
I20260926 03:34:44.793088 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:44.787144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55724 (local address 127.0.97.1:41037)
0926 03:34:44.787373 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:44.787378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:44.787397 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:34:44.787512 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:44.787517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:44.787624 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:44.787776 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:34:44.788220 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:44.789179 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:44.790610 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:44.791094 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:44.791214 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:44.791717 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:34:44.792385 (+ 668us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:34:44.792513 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:44.792670 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 03:34:44.792753 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 03:34:44.803974 14667 raft_consensus.cc:1261] T 660703b9ca474be282d6118fa64cbc25 P cb9ebb92eee54c939139cef3b151f9c9 [term 1 FOLLOWER]: Refusing update from remote peer 3511b88ddb2643189dceaea58e35ce9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:34:44.803994 14533 raft_consensus.cc:1261] T 660703b9ca474be282d6118fa64cbc25 P 1cff45dfdc4c48bc9decf5f844982927 [term 1 FOLLOWER]: Refusing update from remote peer 3511b88ddb2643189dceaea58e35ce9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:34:44.805167 14742 consensus_queue.cc:1035] T 660703b9ca474be282d6118fa64cbc25 P 3511b88ddb2643189dceaea58e35ce9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cff45dfdc4c48bc9decf5f844982927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:34:44.805540 14738 consensus_queue.cc:1035] T 660703b9ca474be282d6118fa64cbc25 P 3511b88ddb2643189dceaea58e35ce9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb9ebb92eee54c939139cef3b151f9c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:34:44.819550 14745 mvcc.cc:205] Tried to move back new op lower bound from 7333452532944760832 to 7333452532709531648. Current Snapshot: MvccSnapshot[applied={T|T < 7333452532944760832}]
I20260926 03:34:44.838151 14255 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:34:44.838546 14255 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:34:44.842748 14255 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514: Sending DeleteTablet for 3 replicas of tablet 660703b9ca474be282d6118fa64cbc25
I20260926 03:34:44.844208 14379 tablet_service.cc:1514] Processing DeleteTablet for tablet 660703b9ca474be282d6118fa64cbc25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:34:44 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55710
I20260926 03:34:44.844201 14647 tablet_service.cc:1514] Processing DeleteTablet for tablet 660703b9ca474be282d6118fa64cbc25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:34:44 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37402
I20260926 03:34:44.844213 14513 tablet_service.cc:1514] Processing DeleteTablet for tablet 660703b9ca474be282d6118fa64cbc25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:34:44 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45628
I20260926 03:34:44.845937 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 14316
I20260926 03:34:44.877509 14756 tablet_replica.cc:330] T 660703b9ca474be282d6118fa64cbc25 P cb9ebb92eee54c939139cef3b151f9c9: stopping tablet replica
I20260926 03:34:44.878515 14756 raft_consensus.cc:2229] T 660703b9ca474be282d6118fa64cbc25 P cb9ebb92eee54c939139cef3b151f9c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:34:44.879889 14756 raft_consensus.cc:2258] T 660703b9ca474be282d6118fa64cbc25 P cb9ebb92eee54c939139cef3b151f9c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:34:44.888140 14756 ts_tablet_manager.cc:1905] T 660703b9ca474be282d6118fa64cbc25 P cb9ebb92eee54c939139cef3b151f9c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:34:44.891755 14754 tablet_replica.cc:330] T 660703b9ca474be282d6118fa64cbc25 P 3511b88ddb2643189dceaea58e35ce9d: stopping tablet replica
I20260926 03:34:44.892449 14755 tablet_replica.cc:330] T 660703b9ca474be282d6118fa64cbc25 P 1cff45dfdc4c48bc9decf5f844982927: stopping tablet replica
I20260926 03:34:44.893069 14755 raft_consensus.cc:2229] T 660703b9ca474be282d6118fa64cbc25 P 1cff45dfdc4c48bc9decf5f844982927 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:34:44.893785 14755 raft_consensus.cc:2258] T 660703b9ca474be282d6118fa64cbc25 P 1cff45dfdc4c48bc9decf5f844982927 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:34:44.896459 14756 ts_tablet_manager.cc:1918] T 660703b9ca474be282d6118fa64cbc25 P cb9ebb92eee54c939139cef3b151f9c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:34:44.896823 14756 log.cc:1196] T 660703b9ca474be282d6118fa64cbc25 P cb9ebb92eee54c939139cef3b151f9c9: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/wal/wals/660703b9ca474be282d6118fa64cbc25
I20260926 03:34:44.897470 14756 ts_tablet_manager.cc:1939] T 660703b9ca474be282d6118fa64cbc25 P cb9ebb92eee54c939139cef3b151f9c9: Deleting consensus metadata
I20260926 03:34:44.897923 14754 raft_consensus.cc:2229] T 660703b9ca474be282d6118fa64cbc25 P 3511b88ddb2643189dceaea58e35ce9d [term 1 LEADER]: Raft consensus shutting down.
Sep 26 03:34:44 dist-test-slave-dt7p krb5kdc[14210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:34:44.898761 14754 raft_consensus.cc:2258] T 660703b9ca474be282d6118fa64cbc25 P 3511b88ddb2643189dceaea58e35ce9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:34:44.899806 14239 catalog_manager.cc:4841] TS cb9ebb92eee54c939139cef3b151f9c9 (127.0.97.3:45531): tablet 660703b9ca474be282d6118fa64cbc25 (table test-table [id=1c62ae4fcb7942c6a95eb9bc00993f7e]) successfully deleted
I20260926 03:34:44.901373 14755 ts_tablet_manager.cc:1905] T 660703b9ca474be282d6118fa64cbc25 P 1cff45dfdc4c48bc9decf5f844982927: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:34:44.903774 14754 ts_tablet_manager.cc:1905] T 660703b9ca474be282d6118fa64cbc25 P 3511b88ddb2643189dceaea58e35ce9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:34:44.909039 14755 ts_tablet_manager.cc:1918] T 660703b9ca474be282d6118fa64cbc25 P 1cff45dfdc4c48bc9decf5f844982927: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:34:44.909216 14755 log.cc:1196] T 660703b9ca474be282d6118fa64cbc25 P 1cff45dfdc4c48bc9decf5f844982927: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/wal/wals/660703b9ca474be282d6118fa64cbc25
I20260926 03:34:44.909816 14755 ts_tablet_manager.cc:1939] T 660703b9ca474be282d6118fa64cbc25 P 1cff45dfdc4c48bc9decf5f844982927: Deleting consensus metadata
I20260926 03:34:44.910290 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:44.847702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55732 (local address 127.0.97.1:41037)
0926 03:34:44.847957 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:44.847972 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:44.870138 (+ 22166us) server_negotiation.cc:407] Connection header received
0926 03:34:44.870892 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:44.870903 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:44.871046 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:44.871157 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:44.880234 (+ 9077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:44.882094 (+ 1860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:44.885666 (+ 3572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:44.886464 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:44.886575 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:44.900807 (+ 14232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:44.900824 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:44.900844 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:44.900913 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:44.905587 (+ 4674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:44.906712 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:44.906722 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:44.906726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:44.906921 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:44.907496 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:44.907501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:44.907505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:44.908018 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:44.908749 (+ 731us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:44.909523 (+ 774us) server_negotiation.cc:299] Negotiation successful
0926 03:34:44.909839 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 03:34:44.912209 14439 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:34:44.912182 14240 catalog_manager.cc:4841] TS 1cff45dfdc4c48bc9decf5f844982927 (127.0.97.2:35639): tablet 660703b9ca474be282d6118fa64cbc25 (table test-table [id=1c62ae4fcb7942c6a95eb9bc00993f7e]) successfully deleted
I20260926 03:34:44.912307 14754 ts_tablet_manager.cc:1918] T 660703b9ca474be282d6118fa64cbc25 P 3511b88ddb2643189dceaea58e35ce9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:34:44.912485 14754 log.cc:1196] T 660703b9ca474be282d6118fa64cbc25 P 3511b88ddb2643189dceaea58e35ce9d: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/wal/wals/660703b9ca474be282d6118fa64cbc25
I20260926 03:34:45.024271 14754 ts_tablet_manager.cc:1939] T 660703b9ca474be282d6118fa64cbc25 P 3511b88ddb2643189dceaea58e35ce9d: Deleting consensus metadata
I20260926 03:34:45.024363 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 14316
I20260926 03:34:45.046737 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 14450
W20260926 03:34:45.046952 14240 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:41037 (error 108)
I20260926 03:34:45.049576 14760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:45.048556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41037 (local address 127.0.0.1:55746)
0926 03:34:45.049124 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:45.049276 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":225,"threads_started":1}
W20260926 03:34:45.049872 14240 catalog_manager.cc:4584] TS 3511b88ddb2643189dceaea58e35ce9d (127.0.97.1:41037): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 660703b9ca474be282d6118fa64cbc25: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41037: connect: Connection refused (error 111)
Sep 26 03:34:45 dist-test-slave-dt7p krb5kdc[14210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:34:45.069067 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:45.047732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45638 (local address 127.0.97.2:35639)
0926 03:34:45.047963 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:45.047972 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:45.048044 (+ 72us) server_negotiation.cc:407] Connection header received
0926 03:34:45.048290 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:45.048301 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:45.048391 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:45.048509 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:45.050345 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:45.051201 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:45.052584 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:45.053018 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:45.053172 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:45.062736 (+ 9564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:45.062758 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:45.062780 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:45.062834 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:45.065882 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:45.066640 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:45.066649 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:45.066652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:45.066812 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:45.067187 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:45.067193 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:45.067196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:45.067649 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:45.067987 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:45.068338 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 03:34:45.068634 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 03:34:45.069871 14573 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:34:45.103888 14760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:45.103230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41037 (local address 127.0.0.1:55752)
0926 03:34:45.103527 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:45.103667 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260926 03:34:45.105309 14760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:45.104765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41037 (local address 127.0.0.1:55758)
0926 03:34:45.105002 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:45.105095 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 03:34:45.166635 14760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:45.166043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41037 (local address 127.0.0.1:55772)
0926 03:34:45.166283 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:45.166437 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 03:34:45.167270 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 14450
I20260926 03:34:45.168046 14760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:45.167508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41037 (local address 127.0.0.1:55782)
0926 03:34:45.167748 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:45.167849 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260926 03:34:45.187021 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 14584
Sep 26 03:34:45 dist-test-slave-dt7p krb5kdc[14210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:34:45.204357 14728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:45.187986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37412 (local address 127.0.97.3:45531)
0926 03:34:45.188215 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:45.188220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:45.188243 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:34:45.188403 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:45.188407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:45.188500 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:45.188663 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:45.189968 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:45.190725 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:45.192048 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:45.192461 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:45.192694 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:45.197389 (+ 4695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:45.197411 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:45.197432 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:45.197477 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:45.201271 (+ 3794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:45.202055 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:45.202065 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:45.202067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:45.202293 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:45.202658 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:45.202662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:45.202664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:45.203085 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:45.203409 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:45.203670 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 03:34:45.203873 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 03:34:45.205224 14707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:34:45.249744 14760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:45.249059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41037 (local address 127.0.0.1:55798)
0926 03:34:45.249360 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:45.249498 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260926 03:34:45.251096 14760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:45.250553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41037 (local address 127.0.0.1:55814)
0926 03:34:45.250801 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:45.250895 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260926 03:34:45.305569 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 14584
I20260926 03:34:45.325248 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 14223
I20260926 03:34:45.326627 14285 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:34:45.457433 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 14223
I20260926 03:34:45.483771 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46679
--webserver_interface=127.0.97.62
--webserver_port=35667
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:34:45.868848 14764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:34:45.869237 14764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:34:45.869408 14764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:34:45.877036 14764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:34:45.877192 14764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:34:45.877297 14764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:34:45.877403 14764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:34:45.877480 14764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:34:45.887887 14764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=35667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:34:45.889708 14764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:34:45.891546 14764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:34:45.921737 14770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:45.921706 14771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:45.921770 14773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:34:45.926369 14764 server_base.cc:1034] running on GCE node
I20260926 03:34:45.927445 14764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:34:45.929502 14764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:34:45.930788 14764 hybrid_clock.cc:648] HybridClock initialized: now 1790393685930728 us; error 53 us; skew 500 ppm
I20260926 03:34:45.931209 14764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:34:45 dist-test-slave-dt7p krb5kdc[14210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:34:45.935240 14764 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:34:45.937228 14764 webserver.cc:460] Webserver started at http://127.0.97.62:35667/ using document root <none> and password file <none>
I20260926 03:34:45.938110 14764 fs_manager.cc:359] Metadata directory not provided
I20260926 03:34:45.938244 14764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:34:45.945168 14764 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 03:34:45.948511 14780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:45.950325 14764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:34:45.950529 14764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "57587beacbcd425cb35e2e05c483e514"
format_stamp: "Formatted at 2026-09-26 03:34:42 on dist-test-slave-dt7p"
server_key: "024b68f856ef40629c1bb4e9181d454e"
server_key_iv: "a59f70e151481a7f3ee2bb49d7763389"
server_key_version: "encryptionkey@0"
I20260926 03:34:45.951787 14764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:34:45.970495 14764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:34:45.975791 14764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:34:45.976145 14764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:34:45.997939 14764 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46679
I20260926 03:34:45.997959 14831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46679 every 8 connection(s)
I20260926 03:34:45.999800 14764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:34:46.001626 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 14764
I20260926 03:34:46.002718 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:41037
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46679
--webserver_port=35465
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:34:46.011691 14832 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:34:46.016326 14832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514: Bootstrap starting.
I20260926 03:34:46.040246 14832 log.cc:824] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514: Log is configured to *not* fsync() on all Append() calls
I20260926 03:34:46.053707 14832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:34:46.054353 14832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514: Bootstrap complete.
I20260926 03:34:46.064713 14832 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57587beacbcd425cb35e2e05c483e514" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46679 } }
I20260926 03:34:46.065733 14832 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57587beacbcd425cb35e2e05c483e514, State: Initialized, Role: FOLLOWER
I20260926 03:34:46.066478 14832 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57587beacbcd425cb35e2e05c483e514" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46679 } }
I20260926 03:34:46.066744 14832 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:34:46.066869 14832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:34:46.067058 14832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [term 1 FOLLOWER]: Advancing to term 2
I20260926 03:34:46.071147 14832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57587beacbcd425cb35e2e05c483e514" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46679 } }
I20260926 03:34:46.071727 14832 leader_election.cc:302] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57587beacbcd425cb35e2e05c483e514; no voters:
I20260926 03:34:46.072551 14832 leader_election.cc:288] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 03:34:46.072739 14836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 03:34:46.075341 14832 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:34:46.079943 14836 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [term 2 LEADER]: Becoming Leader. State: Replica: 57587beacbcd425cb35e2e05c483e514, State: Running, Role: LEADER
I20260926 03:34:46.080714 14836 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57587beacbcd425cb35e2e05c483e514" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46679 } }
I20260926 03:34:46.084201 14838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57587beacbcd425cb35e2e05c483e514. Latest consensus state: current_term: 2 leader_uuid: "57587beacbcd425cb35e2e05c483e514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57587beacbcd425cb35e2e05c483e514" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46679 } } }
I20260926 03:34:46.084720 14838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [sys.catalog]: This master's current role is: LEADER
I20260926 03:34:46.083801 14837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "57587beacbcd425cb35e2e05c483e514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57587beacbcd425cb35e2e05c483e514" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46679 } } }
I20260926 03:34:46.085697 14837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [sys.catalog]: This master's current role is: LEADER
I20260926 03:34:46.098296 14848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:34:46.107370 14848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:34:46.109337 14848 catalog_manager.cc:1259] Loaded cluster ID: 04964e8c9e844d89b322d8c905759b1d
I20260926 03:34:46.109520 14848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:34:46.113874 14848 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 03:34:46.129536 14848 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 03:34:46.137804 14848 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:34:46.139274 14848 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514: Loaded TSK: 0
I20260926 03:34:46.140533 14848 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 03:34:46.434185 14834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:34:46.434546 14834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:34:46.434711 14834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:34:46.442968 14834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:34:46.443125 14834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:34:46.443323 14834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:34:46.454656 14834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:41037
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=35465
--tserver_master_addrs=127.0.97.62:46679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:34:46.456422 14834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:34:46.458101 14834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:34:46.478322 14860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:46.480957 14861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:46.485963 14863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:34:46.486723 14834 server_base.cc:1034] running on GCE node
I20260926 03:34:46.487551 14834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:34:46.488853 14834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:34:46.490099 14834 hybrid_clock.cc:648] HybridClock initialized: now 1790393686490046 us; error 41 us; skew 500 ppm
I20260926 03:34:46.490496 14834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:34:46 dist-test-slave-dt7p krb5kdc[14210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:34:46.494210 14834 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:34:46.496244 14834 webserver.cc:460] Webserver started at http://127.0.97.1:35465/ using document root <none> and password file <none>
I20260926 03:34:46.497192 14834 fs_manager.cc:359] Metadata directory not provided
I20260926 03:34:46.497361 14834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:34:46.503926 14834 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 03:34:46.507316 14870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:46.509086 14834 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 03:34:46.509287 14834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/wal
uuid: "3511b88ddb2643189dceaea58e35ce9d"
format_stamp: "Formatted at 2026-09-26 03:34:43 on dist-test-slave-dt7p"
server_key: "cc8b2b244a878c26f943c89be916d847"
server_key_iv: "1301223c8b27074a4b02588290d98e64"
server_key_version: "encryptionkey@0"
I20260926 03:34:46.510453 14834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:34:46.528792 14834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:34:46.533993 14834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:34:46.534312 14834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:34:46.535449 14834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:34:46.537034 14834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:34:46.537155 14834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:46.537288 14834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:34:46.537375 14834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:46.573835 14834 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41037
I20260926 03:34:46.573885 14982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41037 every 8 connection(s)
I20260926 03:34:46.575595 14834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:34:46.585634 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 14834
I20260926 03:34:46.586362 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:35639
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46679
--webserver_port=40145
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:34:46 dist-test-slave-dt7p krb5kdc[14210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:34:46.604488 14986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:46.579212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.1:43337)
0926 03:34:46.580173 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:46.580230 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 03:34:46.582248 (+ 2018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:34:46.583828 (+ 1580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:34:46.583836 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:34:46.584408 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:34:46.585264 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:46.585293 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:46.587382 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:46.587390 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:46.589784 (+ 2394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:46.589812 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:46.590984 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:46.590993 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:46.591336 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:46.592278 (+ 942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:34:46.592315 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:34:46.595350 (+ 3035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:34:46.599575 (+ 4225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:46.599585 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:46.599601 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:46.600043 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:46.600514 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:46.600520 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:46.600523 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:46.600791 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:46.601591 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:34:46.601599 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:34:46.601982 (+ 383us) client_negotiation.cc:769] Sending connection context
0926 03:34:46.602276 (+ 294us) client_negotiation.cc:240] Negotiation successful
0926 03:34:46.602918 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":402,"threads_started":1}
I20260926 03:34:46.604483 14985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:46.578907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43337 (local address 127.0.97.62:46679)
0926 03:34:46.579545 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:46.579558 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:46.580359 (+ 801us) server_negotiation.cc:407] Connection header received
0926 03:34:46.582852 (+ 2493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:46.582866 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:46.583268 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:46.583600 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:46.585573 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:46.587193 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:46.589985 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:46.590709 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:46.590876 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:46.595716 (+ 4840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:46.595731 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:46.595766 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:46.595831 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:46.599408 (+ 3577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:46.600202 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:46.600209 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:46.600214 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:46.600394 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:46.600912 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:46.600921 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:46.600923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:46.601451 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:46.601672 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:46.602505 (+ 833us) server_negotiation.cc:299] Negotiation successful
0926 03:34:46.602939 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":289,"threads_started":1}
I20260926 03:34:46.607966 14983 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46679
I20260926 03:34:46.608410 14983 heartbeater.cc:464] Registering TS with master...
I20260926 03:34:46.609619 14983 heartbeater.cc:511] Master 127.0.97.62:46679 requested a full tablet report, sending...
I20260926 03:34:46.612515 14797 ts_manager.cc:194] Registered new tserver with Master: 3511b88ddb2643189dceaea58e35ce9d (127.0.97.1:41037)
I20260926 03:34:46.614293 14797 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43337
W20260926 03:34:46.972935 14987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:34:46.973333 14987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:34:46.973506 14987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:34:46.981779 14987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:34:46.981933 14987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:34:46.982120 14987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:34:46.992929 14987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:35639
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=40145
--tserver_master_addrs=127.0.97.62:46679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:34:46.994586 14987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:34:46.996170 14987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:34:47.013504 14992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:47.013621 14993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:47.013520 14995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:34:47.014499 14987 server_base.cc:1034] running on GCE node
I20260926 03:34:47.015308 14987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:34:47.016623 14987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:34:47.017871 14987 hybrid_clock.cc:648] HybridClock initialized: now 1790393687017824 us; error 41 us; skew 500 ppm
I20260926 03:34:47.018281 14987 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:34:47 dist-test-slave-dt7p krb5kdc[14210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:34:47.022284 14987 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:34:47.023965 14987 webserver.cc:460] Webserver started at http://127.0.97.2:40145/ using document root <none> and password file <none>
I20260926 03:34:47.024873 14987 fs_manager.cc:359] Metadata directory not provided
I20260926 03:34:47.025028 14987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:34:47.031400 14987 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 03:34:47.034672 15002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:47.036160 14987 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:34:47.036358 14987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/wal
uuid: "1cff45dfdc4c48bc9decf5f844982927"
format_stamp: "Formatted at 2026-09-26 03:34:43 on dist-test-slave-dt7p"
server_key: "8a1a4c57ef90077cdce1ecdd3e73f827"
server_key_iv: "5afae4554131e9a0c80d726fa118248e"
server_key_version: "encryptionkey@0"
I20260926 03:34:47.037525 14987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:34:47.056223 14987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:34:47.060787 14987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:34:47.061097 14987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:34:47.062194 14987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:34:47.063665 14987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:34:47.063757 14987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:47.063920 14987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:34:47.064030 14987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:47.101866 14987 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35639
I20260926 03:34:47.101939 15114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35639 every 8 connection(s)
I20260926 03:34:47.104058 14987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:34:47.107585 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 14987
I20260926 03:34:47.108415 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:45531
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46679
--webserver_port=38903
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:34:47 dist-test-slave-dt7p krb5kdc[14210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:34:47.128718 15117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:47.107448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57123 (local address 127.0.97.62:46679)
0926 03:34:47.108053 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:47.108062 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:47.109243 (+ 1181us) server_negotiation.cc:407] Connection header received
0926 03:34:47.111377 (+ 2134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:47.111387 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:47.111518 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:47.111678 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:47.113492 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:47.114742 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:47.116686 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:47.117189 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:47.117311 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:47.122336 (+ 5025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:47.122342 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:47.122345 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:47.122386 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:47.125419 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:47.126186 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:47.126196 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:47.126198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:47.126336 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:47.126781 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:47.126791 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:47.126793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:47.127079 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:47.127257 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:47.128001 (+ 744us) server_negotiation.cc:299] Negotiation successful
0926 03:34:47.128253 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":288,"threads_started":1}
I20260926 03:34:47.129926 15118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:47.108094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.2:57123)
0926 03:34:47.109048 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:47.109099 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 03:34:47.111046 (+ 1947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:34:47.111922 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:34:47.111931 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:34:47.112522 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:34:47.113284 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:47.113304 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:47.114907 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:47.114911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:47.116496 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:47.116509 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:47.117446 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:47.117454 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:47.117807 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:47.118681 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:34:47.118719 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:34:47.122046 (+ 3327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:34:47.125572 (+ 3526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:47.125583 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:47.125603 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:47.126051 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:47.126445 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:47.126451 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:47.126455 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:47.126660 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:47.127210 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:34:47.127219 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:34:47.127617 (+ 398us) client_negotiation.cc:769] Sending connection context
0926 03:34:47.127906 (+ 289us) client_negotiation.cc:240] Negotiation successful
0926 03:34:47.128553 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":425,"threads_started":1}
I20260926 03:34:47.131803 15115 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46679
I20260926 03:34:47.132179 15115 heartbeater.cc:464] Registering TS with master...
I20260926 03:34:47.133114 15115 heartbeater.cc:511] Master 127.0.97.62:46679 requested a full tablet report, sending...
I20260926 03:34:47.134914 14797 ts_manager.cc:194] Registered new tserver with Master: 1cff45dfdc4c48bc9decf5f844982927 (127.0.97.2:35639)
I20260926 03:34:47.135872 14797 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57123
W20260926 03:34:47.491572 15119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:34:47.491885 15119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:34:47.491988 15119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:34:47.499845 15119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:34:47.499979 15119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:34:47.500108 15119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:34:47.510972 15119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:45531
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=38903
--tserver_master_addrs=127.0.97.62:46679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:34:47.512708 15119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:34:47.514322 15119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:34:47.530956 15124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:47.530918 15127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:47.530998 15125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:34:47.532269 15119 server_base.cc:1034] running on GCE node
I20260926 03:34:47.533079 15119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:34:47.534207 15119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:34:47.535444 15119 hybrid_clock.cc:648] HybridClock initialized: now 1790393687535383 us; error 54 us; skew 500 ppm
I20260926 03:34:47.535844 15119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:34:47 dist-test-slave-dt7p krb5kdc[14210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:34:47.539500 15119 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:34:47.541495 15119 webserver.cc:460] Webserver started at http://127.0.97.3:38903/ using document root <none> and password file <none>
I20260926 03:34:47.542421 15119 fs_manager.cc:359] Metadata directory not provided
I20260926 03:34:47.542593 15119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:34:47.549002 15119 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 03:34:47.552211 15134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:47.553954 15119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:34:47.554180 15119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/wal
uuid: "cb9ebb92eee54c939139cef3b151f9c9"
format_stamp: "Formatted at 2026-09-26 03:34:44 on dist-test-slave-dt7p"
server_key: "6148b146126b8f37a04fe7a627f95e41"
server_key_iv: "bae1a42b5a61105223a88eeeb1c898fa"
server_key_version: "encryptionkey@0"
I20260926 03:34:47.555404 15119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:34:47.578408 15119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:34:47.583940 15119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:34:47.584211 15119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:34:47.585450 15119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:34:47.586985 15119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:34:47.587088 15119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:47.587245 15119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:34:47.587317 15119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:47.618219 14983 heartbeater.cc:503] Master 127.0.97.62:46679 was elected leader, sending a full tablet report...
I20260926 03:34:47.625145 15119 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45531
I20260926 03:34:47.625201 15246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45531 every 8 connection(s)
I20260926 03:34:47.626895 15119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:34:47.628501 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 15119
Sep 26 03:34:47 dist-test-slave-dt7p krb5kdc[14210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:34:47.650731 15250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:47.629868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38632 (local address 127.0.97.62:46679)
0926 03:34:47.634051 (+ 4183us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:47.634058 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:47.634086 (+ 28us) server_negotiation.cc:407] Connection header received
0926 03:34:47.634167 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:47.634186 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:47.634298 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:47.634405 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:47.636743 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:47.637583 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:47.639158 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:47.639588 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:47.639685 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:47.643583 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:47.643588 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:47.643592 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:47.643627 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:47.647571 (+ 3944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:47.648427 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:47.648435 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:47.648439 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:47.648694 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:47.649099 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:47.649104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:47.649108 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:47.649479 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:47.649777 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:47.650170 (+ 393us) server_negotiation.cc:299] Negotiation successful
0926 03:34:47.650381 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4001,"thread_start_us":3852,"threads_started":1}
I20260926 03:34:47.653838 15252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:47.632187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34781 (local address 127.0.97.62:46679)
0926 03:34:47.634197 (+ 2010us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:47.634209 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:47.634224 (+ 15us) server_negotiation.cc:407] Connection header received
0926 03:34:47.636001 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:47.636005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:47.636059 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:47.636162 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:47.638105 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:47.638895 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:47.641123 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:47.641663 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:47.641777 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:47.646404 (+ 4627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:47.646414 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:47.646426 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:47.646499 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:47.649786 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:47.650821 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:47.650830 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:47.650832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:47.650985 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:47.651555 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:47.651560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:47.651565 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:47.651991 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:47.652155 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:47.653248 (+ 1093us) server_negotiation.cc:299] Negotiation successful
0926 03:34:47.653436 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1888,"thread_start_us":1759,"threads_started":1}
I20260926 03:34:47.655316 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:47.631431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.3:34781)
0926 03:34:47.632586 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:47.632662 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0926 03:34:47.635640 (+ 2978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:34:47.636494 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:34:47.636507 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:34:47.637153 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:34:47.637929 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:47.637954 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:47.639058 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:47.639064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:47.640970 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:47.640989 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:47.641940 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:47.641949 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:47.642306 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:47.643043 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:34:47.643083 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:34:47.646108 (+ 3025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:34:47.649937 (+ 3829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:47.649947 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:47.649970 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:47.650635 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:47.651113 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:47.651118 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:47.651122 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:47.651400 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:47.652193 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:34:47.652200 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:34:47.652747 (+ 547us) client_negotiation.cc:769] Sending connection context
0926 03:34:47.653145 (+ 398us) client_negotiation.cc:240] Negotiation successful
0926 03:34:47.653822 (+ 677us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":788,"threads_started":1}
I20260926 03:34:47.657128 15247 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46679
I20260926 03:34:47.657500 15247 heartbeater.cc:464] Registering TS with master...
I20260926 03:34:47.658372 15247 heartbeater.cc:511] Master 127.0.97.62:46679 requested a full tablet report, sending...
I20260926 03:34:47.660208 14797 ts_manager.cc:194] Registered new tserver with Master: cb9ebb92eee54c939139cef3b151f9c9 (127.0.97.3:45531)
I20260926 03:34:47.661329 14797 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34781
I20260926 03:34:47.667187 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:34:47.674993 15252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:47.668378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38640 (local address 127.0.97.62:46679)
0926 03:34:47.668635 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:47.668640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:47.669068 (+ 428us) server_negotiation.cc:407] Connection header received
0926 03:34:47.669266 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:47.669270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:47.669438 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:47.669574 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:34:47.670001 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:47.670770 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:47.672546 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:47.673064 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:47.673192 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:47.673747 (+ 555us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:34:47.674433 (+ 686us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:34:47.674567 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:47.674659 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 03:34:47.674741 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 03:34:47.676189 14797 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:38640:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:34:47.684715 14797 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:34:47.722599 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:47.700331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37420 (local address 127.0.97.3:45531)
0926 03:34:47.710125 (+ 9794us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:47.710139 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:47.710185 (+ 46us) server_negotiation.cc:407] Connection header received
0926 03:34:47.710330 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:47.710341 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:47.710666 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:47.710936 (+ 270us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:34:47.712202 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:47.714525 (+ 2323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:47.718979 (+ 4454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:47.720664 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:47.720777 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:47.721046 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:47.721929 (+ 883us) server_negotiation.cc:299] Negotiation successful
0926 03:34:47.722199 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9501,"thread_start_us":15179,"threads_started":1}
I20260926 03:34:47.722656 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:47.700047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45531 (local address 127.0.0.1:37420)
0926 03:34:47.709211 (+ 9164us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:47.709235 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:34:47.709392 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:34:47.711338 (+ 1946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:34:47.711346 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:34:47.711382 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:34:47.711916 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:47.711932 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:47.714773 (+ 2841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:47.714783 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:47.718741 (+ 3958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:47.718757 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:47.721289 (+ 2532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:47.721298 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:47.721598 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:47.721677 (+ 79us) client_negotiation.cc:769] Sending connection context
0926 03:34:47.721842 (+ 165us) client_negotiation.cc:240] Negotiation successful
0926 03:34:47.722160 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8987,"mutex_wait_us":881,"thread_start_us":6719,"threads_started":1}
I20260926 03:34:47.722707 15258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:47.699101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45650 (local address 127.0.97.2:35639)
0926 03:34:47.705616 (+ 6515us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:47.705629 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:47.707942 (+ 2313us) server_negotiation.cc:407] Connection header received
0926 03:34:47.708113 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:47.708126 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:47.708532 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:47.708817 (+ 285us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:34:47.711531 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:47.715697 (+ 4166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:47.719912 (+ 4215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:47.721140 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:47.721299 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:47.721538 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:47.722152 (+ 614us) server_negotiation.cc:299] Negotiation successful
0926 03:34:47.722353 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6320,"thread_start_us":6172,"threads_started":1}
I20260926 03:34:47.724598 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:47.699800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35639 (local address 127.0.0.1:45650)
0926 03:34:47.707321 (+ 7521us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:47.707383 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 03:34:47.707660 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:34:47.710209 (+ 2549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:34:47.710222 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:34:47.710297 (+ 75us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:34:47.710957 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:47.710982 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:47.715961 (+ 4979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:47.715970 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:47.719692 (+ 3722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:47.719706 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:47.721476 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:47.721483 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:47.721658 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:47.721682 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 03:34:47.721842 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 03:34:47.724145 (+ 2303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7117,"mutex_wait_us":955,"thread_start_us":9009,"threads_started":1}
I20260926 03:34:47.726766 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:47.709575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41037 (local address 127.0.0.1:55828)
0926 03:34:47.717653 (+ 8078us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:47.717674 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:34:47.717813 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:34:47.718570 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:34:47.718575 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:34:47.718605 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:34:47.719118 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:47.719130 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:47.721180 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:47.721187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:47.723558 (+ 2371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:47.723571 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:47.725858 (+ 2287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:47.725864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:47.726044 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:47.726086 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 03:34:47.726209 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 03:34:47.726345 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7922,"mutex_wait_us":1,"thread_start_us":7710,"threads_started":1}
I20260926 03:34:47.726815 15263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:47.709679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55828 (local address 127.0.97.1:41037)
0926 03:34:47.717260 (+ 7581us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:47.717274 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:47.717808 (+ 534us) server_negotiation.cc:407] Connection header received
0926 03:34:47.717995 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:47.718005 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:47.718332 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:47.718578 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:34:47.719348 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:47.721031 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:47.723742 (+ 2711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:47.725316 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:47.725447 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:47.725674 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:47.726269 (+ 595us) server_negotiation.cc:299] Negotiation successful
0926 03:34:47.726452 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7363,"thread_start_us":7168,"threads_started":1}
I20260926 03:34:47.728042 15182 tablet_service.cc:1467] Processing CreateTablet for tablet 959366c593174546ba571f738626f85a (DEFAULT_TABLE table=test-table [id=5331724f01b146c6a2fd87c92c48406f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:34:47.728847 15050 tablet_service.cc:1467] Processing CreateTablet for tablet 959366c593174546ba571f738626f85a (DEFAULT_TABLE table=test-table [id=5331724f01b146c6a2fd87c92c48406f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:34:47.729965 15182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 959366c593174546ba571f738626f85a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:34:47.730258 15050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 959366c593174546ba571f738626f85a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:34:47.730289 14918 tablet_service.cc:1467] Processing CreateTablet for tablet 959366c593174546ba571f738626f85a (DEFAULT_TABLE table=test-table [id=5331724f01b146c6a2fd87c92c48406f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:34:47.731477 14918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 959366c593174546ba571f738626f85a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:34:47.744316 15264 tablet_bootstrap.cc:492] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d: Bootstrap starting.
I20260926 03:34:47.746582 15265 tablet_bootstrap.cc:492] T 959366c593174546ba571f738626f85a P cb9ebb92eee54c939139cef3b151f9c9: Bootstrap starting.
I20260926 03:34:47.749033 15264 tablet_bootstrap.cc:654] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d: Neither blocks nor log segments found. Creating new log.
I20260926 03:34:47.749338 15266 tablet_bootstrap.cc:492] T 959366c593174546ba571f738626f85a P 1cff45dfdc4c48bc9decf5f844982927: Bootstrap starting.
I20260926 03:34:47.750205 15265 tablet_bootstrap.cc:654] T 959366c593174546ba571f738626f85a P cb9ebb92eee54c939139cef3b151f9c9: Neither blocks nor log segments found. Creating new log.
I20260926 03:34:47.750578 15264 log.cc:824] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d: Log is configured to *not* fsync() on all Append() calls
I20260926 03:34:47.751523 15265 log.cc:824] T 959366c593174546ba571f738626f85a P cb9ebb92eee54c939139cef3b151f9c9: Log is configured to *not* fsync() on all Append() calls
I20260926 03:34:47.752766 15266 tablet_bootstrap.cc:654] T 959366c593174546ba571f738626f85a P 1cff45dfdc4c48bc9decf5f844982927: Neither blocks nor log segments found. Creating new log.
I20260926 03:34:47.754060 15266 log.cc:824] T 959366c593174546ba571f738626f85a P 1cff45dfdc4c48bc9decf5f844982927: Log is configured to *not* fsync() on all Append() calls
I20260926 03:34:47.754266 15264 tablet_bootstrap.cc:492] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d: No bootstrap required, opened a new log
I20260926 03:34:47.754581 15264 ts_tablet_manager.cc:1397] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 03:34:47.755870 15265 tablet_bootstrap.cc:492] T 959366c593174546ba571f738626f85a P cb9ebb92eee54c939139cef3b151f9c9: No bootstrap required, opened a new log
I20260926 03:34:47.756155 15265 ts_tablet_manager.cc:1397] T 959366c593174546ba571f738626f85a P cb9ebb92eee54c939139cef3b151f9c9: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260926 03:34:47.757212 15266 tablet_bootstrap.cc:492] T 959366c593174546ba571f738626f85a P 1cff45dfdc4c48bc9decf5f844982927: No bootstrap required, opened a new log
I20260926 03:34:47.757463 15266 ts_tablet_manager.cc:1397] T 959366c593174546ba571f738626f85a P 1cff45dfdc4c48bc9decf5f844982927: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260926 03:34:47.763093 15264 raft_consensus.cc:348] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff45dfdc4c48bc9decf5f844982927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35639 } } peers { permanent_uuid: "3511b88ddb2643189dceaea58e35ce9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41037 } } peers { permanent_uuid: "cb9ebb92eee54c939139cef3b151f9c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45531 } }
I20260926 03:34:47.763509 15264 raft_consensus.cc:374] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:34:47.763621 15264 raft_consensus.cc:729] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3511b88ddb2643189dceaea58e35ce9d, State: Initialized, Role: FOLLOWER
I20260926 03:34:47.764292 15264 consensus_queue.cc:260] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff45dfdc4c48bc9decf5f844982927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35639 } } peers { permanent_uuid: "3511b88ddb2643189dceaea58e35ce9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41037 } } peers { permanent_uuid: "cb9ebb92eee54c939139cef3b151f9c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45531 } }
I20260926 03:34:47.766240 15264 ts_tablet_manager.cc:1428] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20260926 03:34:47.767427 15265 raft_consensus.cc:348] T 959366c593174546ba571f738626f85a P cb9ebb92eee54c939139cef3b151f9c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff45dfdc4c48bc9decf5f844982927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35639 } } peers { permanent_uuid: "3511b88ddb2643189dceaea58e35ce9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41037 } } peers { permanent_uuid: "cb9ebb92eee54c939139cef3b151f9c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45531 } }
I20260926 03:34:47.767892 15265 raft_consensus.cc:374] T 959366c593174546ba571f738626f85a P cb9ebb92eee54c939139cef3b151f9c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:34:47.767995 15265 raft_consensus.cc:729] T 959366c593174546ba571f738626f85a P cb9ebb92eee54c939139cef3b151f9c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb9ebb92eee54c939139cef3b151f9c9, State: Initialized, Role: FOLLOWER
I20260926 03:34:47.768657 15265 consensus_queue.cc:260] T 959366c593174546ba571f738626f85a P cb9ebb92eee54c939139cef3b151f9c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff45dfdc4c48bc9decf5f844982927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35639 } } peers { permanent_uuid: "3511b88ddb2643189dceaea58e35ce9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41037 } } peers { permanent_uuid: "cb9ebb92eee54c939139cef3b151f9c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45531 } }
I20260926 03:34:47.769867 15247 heartbeater.cc:503] Master 127.0.97.62:46679 was elected leader, sending a full tablet report...
I20260926 03:34:47.770457 15265 ts_tablet_manager.cc:1428] T 959366c593174546ba571f738626f85a P cb9ebb92eee54c939139cef3b151f9c9: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260926 03:34:47.769470 15266 raft_consensus.cc:348] T 959366c593174546ba571f738626f85a P 1cff45dfdc4c48bc9decf5f844982927 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff45dfdc4c48bc9decf5f844982927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35639 } } peers { permanent_uuid: "3511b88ddb2643189dceaea58e35ce9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41037 } } peers { permanent_uuid: "cb9ebb92eee54c939139cef3b151f9c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45531 } }
I20260926 03:34:47.770886 15266 raft_consensus.cc:374] T 959366c593174546ba571f738626f85a P 1cff45dfdc4c48bc9decf5f844982927 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:34:47.771034 15266 raft_consensus.cc:729] T 959366c593174546ba571f738626f85a P 1cff45dfdc4c48bc9decf5f844982927 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cff45dfdc4c48bc9decf5f844982927, State: Initialized, Role: FOLLOWER
I20260926 03:34:47.771684 15266 consensus_queue.cc:260] T 959366c593174546ba571f738626f85a P 1cff45dfdc4c48bc9decf5f844982927 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff45dfdc4c48bc9decf5f844982927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35639 } } peers { permanent_uuid: "3511b88ddb2643189dceaea58e35ce9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41037 } } peers { permanent_uuid: "cb9ebb92eee54c939139cef3b151f9c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45531 } }
I20260926 03:34:47.773834 15115 heartbeater.cc:503] Master 127.0.97.62:46679 was elected leader, sending a full tablet report...
I20260926 03:34:47.774381 15266 ts_tablet_manager.cc:1428] T 959366c593174546ba571f738626f85a P 1cff45dfdc4c48bc9decf5f844982927: Time spent starting tablet: real 0.017s user 0.013s sys 0.002s
W20260926 03:34:47.828346 14984 tablet.cc:2369] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:34:47.856797 15116 tablet.cc:2369] T 959366c593174546ba571f738626f85a P 1cff45dfdc4c48bc9decf5f844982927: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:34:47.878916 15248 tablet.cc:2369] T 959366c593174546ba571f738626f85a P cb9ebb92eee54c939139cef3b151f9c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:34:47.949430 15271 raft_consensus.cc:482] T 959366c593174546ba571f738626f85a P cb9ebb92eee54c939139cef3b151f9c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:34:47.949884 15271 raft_consensus.cc:504] T 959366c593174546ba571f738626f85a P cb9ebb92eee54c939139cef3b151f9c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff45dfdc4c48bc9decf5f844982927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35639 } } peers { permanent_uuid: "3511b88ddb2643189dceaea58e35ce9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41037 } } peers { permanent_uuid: "cb9ebb92eee54c939139cef3b151f9c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45531 } }
I20260926 03:34:47.952170 15271 leader_election.cc:288] T 959366c593174546ba571f738626f85a P cb9ebb92eee54c939139cef3b151f9c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cff45dfdc4c48bc9decf5f844982927 (127.0.97.2:35639), 3511b88ddb2643189dceaea58e35ce9d (127.0.97.1:41037)
I20260926 03:34:47.960067 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:47.952458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35639 (local address 127.0.97.3:36631)
0926 03:34:47.952708 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:47.952731 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:34:47.952890 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:34:47.953535 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:34:47.953541 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:34:47.953575 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:34:47.953840 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:47.953854 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:47.955345 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:47.955349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:47.957564 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:47.957580 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:47.959187 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:47.959191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:47.959390 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:47.959418 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 03:34:47.959495 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 03:34:47.959709 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":5}
I20260926 03:34:47.960111 15258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:47.952821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36631 (local address 127.0.97.2:35639)
0926 03:34:47.953105 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:47.953114 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:47.953142 (+ 28us) server_negotiation.cc:407] Connection header received
0926 03:34:47.953245 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:47.953252 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:47.953353 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:47.953477 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:34:47.953990 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:47.955197 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:47.957753 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:47.959020 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:47.959196 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:47.959322 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:47.959559 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 03:34:47.959731 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 03:34:47.960493 15263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:47.953251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38275 (local address 127.0.97.1:41037)
0926 03:34:47.953515 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:47.953522 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:47.953765 (+ 243us) server_negotiation.cc:407] Connection header received
0926 03:34:47.953991 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:47.953996 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:47.954094 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:47.954203 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:34:47.954723 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:47.955742 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:47.958137 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:47.959447 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:47.959619 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:47.959740 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:47.960052 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 03:34:47.960213 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 03:34:47.960534 15273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:47.952896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41037 (local address 127.0.97.3:38275)
0926 03:34:47.953632 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:47.953657 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 03:34:47.953807 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:34:47.954230 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:34:47.954233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:34:47.954260 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:34:47.954568 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:47.954576 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:47.955899 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:47.955907 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:47.957979 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:47.957989 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:47.959694 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:47.959699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:47.959900 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:47.959932 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 03:34:47.960018 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 03:34:47.960164 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":402,"threads_started":1}
I20260926 03:34:47.961968 15070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959366c593174546ba571f738626f85a" candidate_uuid: "cb9ebb92eee54c939139cef3b151f9c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cff45dfdc4c48bc9decf5f844982927" is_pre_election: true
I20260926 03:34:47.961967 14938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959366c593174546ba571f738626f85a" candidate_uuid: "cb9ebb92eee54c939139cef3b151f9c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3511b88ddb2643189dceaea58e35ce9d" is_pre_election: true
I20260926 03:34:47.962678 15070 raft_consensus.cc:2454] T 959366c593174546ba571f738626f85a P 1cff45dfdc4c48bc9decf5f844982927 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb9ebb92eee54c939139cef3b151f9c9 in term 0.
I20260926 03:34:47.962678 14938 raft_consensus.cc:2454] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb9ebb92eee54c939139cef3b151f9c9 in term 0.
I20260926 03:34:47.963483 15136 leader_election.cc:302] T 959366c593174546ba571f738626f85a P cb9ebb92eee54c939139cef3b151f9c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3511b88ddb2643189dceaea58e35ce9d, cb9ebb92eee54c939139cef3b151f9c9; no voters:
I20260926 03:34:47.963888 15271 raft_consensus.cc:2781] T 959366c593174546ba571f738626f85a P cb9ebb92eee54c939139cef3b151f9c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:34:47.964071 15271 raft_consensus.cc:482] T 959366c593174546ba571f738626f85a P cb9ebb92eee54c939139cef3b151f9c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:34:47.964211 15271 raft_consensus.cc:3037] T 959366c593174546ba571f738626f85a P cb9ebb92eee54c939139cef3b151f9c9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:34:47.967507 15271 raft_consensus.cc:504] T 959366c593174546ba571f738626f85a P cb9ebb92eee54c939139cef3b151f9c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff45dfdc4c48bc9decf5f844982927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35639 } } peers { permanent_uuid: "3511b88ddb2643189dceaea58e35ce9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41037 } } peers { permanent_uuid: "cb9ebb92eee54c939139cef3b151f9c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45531 } }
I20260926 03:34:47.968235 15271 leader_election.cc:288] T 959366c593174546ba571f738626f85a P cb9ebb92eee54c939139cef3b151f9c9 [CANDIDATE]: Term 1 election: Requested vote from peers 1cff45dfdc4c48bc9decf5f844982927 (127.0.97.2:35639), 3511b88ddb2643189dceaea58e35ce9d (127.0.97.1:41037)
I20260926 03:34:47.968830 15070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959366c593174546ba571f738626f85a" candidate_uuid: "cb9ebb92eee54c939139cef3b151f9c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cff45dfdc4c48bc9decf5f844982927"
I20260926 03:34:47.968916 14938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "959366c593174546ba571f738626f85a" candidate_uuid: "cb9ebb92eee54c939139cef3b151f9c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3511b88ddb2643189dceaea58e35ce9d"
I20260926 03:34:47.969086 15070 raft_consensus.cc:3037] T 959366c593174546ba571f738626f85a P 1cff45dfdc4c48bc9decf5f844982927 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:34:47.969183 14938 raft_consensus.cc:3037] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:34:47.971632 15070 raft_consensus.cc:2454] T 959366c593174546ba571f738626f85a P 1cff45dfdc4c48bc9decf5f844982927 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb9ebb92eee54c939139cef3b151f9c9 in term 1.
I20260926 03:34:47.971873 14938 raft_consensus.cc:2454] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb9ebb92eee54c939139cef3b151f9c9 in term 1.
I20260926 03:34:47.972137 15136 leader_election.cc:302] T 959366c593174546ba571f738626f85a P cb9ebb92eee54c939139cef3b151f9c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cff45dfdc4c48bc9decf5f844982927, cb9ebb92eee54c939139cef3b151f9c9; no voters:
I20260926 03:34:47.972470 15271 raft_consensus.cc:2781] T 959366c593174546ba571f738626f85a P cb9ebb92eee54c939139cef3b151f9c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:34:47.972988 15271 raft_consensus.cc:686] T 959366c593174546ba571f738626f85a P cb9ebb92eee54c939139cef3b151f9c9 [term 1 LEADER]: Becoming Leader. State: Replica: cb9ebb92eee54c939139cef3b151f9c9, State: Running, Role: LEADER
I20260926 03:34:47.973559 15271 consensus_queue.cc:237] T 959366c593174546ba571f738626f85a P cb9ebb92eee54c939139cef3b151f9c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff45dfdc4c48bc9decf5f844982927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35639 } } peers { permanent_uuid: "3511b88ddb2643189dceaea58e35ce9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41037 } } peers { permanent_uuid: "cb9ebb92eee54c939139cef3b151f9c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45531 } }
I20260926 03:34:47.979336 14797 catalog_manager.cc:5495] T 959366c593174546ba571f738626f85a P cb9ebb92eee54c939139cef3b151f9c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb9ebb92eee54c939139cef3b151f9c9 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "cb9ebb92eee54c939139cef3b151f9c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff45dfdc4c48bc9decf5f844982927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3511b88ddb2643189dceaea58e35ce9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb9ebb92eee54c939139cef3b151f9c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45531 } health_report { overall_health: HEALTHY } } }
I20260926 03:34:48.014453 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:48.007617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37428 (local address 127.0.97.3:45531)
0926 03:34:48.007874 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:48.007881 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:48.007979 (+ 98us) server_negotiation.cc:407] Connection header received
0926 03:34:48.008309 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:48.008315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:48.008468 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:48.008668 (+ 200us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:34:48.009115 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:48.009926 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:48.012012 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:48.012432 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:48.012528 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:48.013141 (+ 613us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:34:48.013811 (+ 670us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:34:48.013926 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:48.014094 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 03:34:48.014172 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 03:34:48.024416 14938 raft_consensus.cc:1261] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [term 1 FOLLOWER]: Refusing update from remote peer cb9ebb92eee54c939139cef3b151f9c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:34:48.024416 15070 raft_consensus.cc:1261] T 959366c593174546ba571f738626f85a P 1cff45dfdc4c48bc9decf5f844982927 [term 1 FOLLOWER]: Refusing update from remote peer cb9ebb92eee54c939139cef3b151f9c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:34:48.025494 15271 consensus_queue.cc:1035] T 959366c593174546ba571f738626f85a P cb9ebb92eee54c939139cef3b151f9c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3511b88ddb2643189dceaea58e35ce9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:34:48.025859 15274 consensus_queue.cc:1035] T 959366c593174546ba571f738626f85a P cb9ebb92eee54c939139cef3b151f9c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cff45dfdc4c48bc9decf5f844982927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:34:48.038861 15278 mvcc.cc:205] Tried to move back new op lower bound from 7333452546136064000 to 7333452545944018944. Current Snapshot: MvccSnapshot[applied={T|T < 7333452546136064000}]
I20260926 03:34:48.057129 14796 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:38640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:34:48.057520 14796 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:38640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:34:48.061980 14796 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514: Sending DeleteTablet for 3 replicas of tablet 959366c593174546ba571f738626f85a
I20260926 03:34:48.063189 15050 tablet_service.cc:1514] Processing DeleteTablet for tablet 959366c593174546ba571f738626f85a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:34:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45650
I20260926 03:34:48.063570 14918 tablet_service.cc:1514] Processing DeleteTablet for tablet 959366c593174546ba571f738626f85a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:34:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55828
I20260926 03:34:48.063980 15182 tablet_service.cc:1514] Processing DeleteTablet for tablet 959366c593174546ba571f738626f85a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:34:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37420
I20260926 03:34:48.064193 15287 tablet_replica.cc:330] T 959366c593174546ba571f738626f85a P 1cff45dfdc4c48bc9decf5f844982927: stopping tablet replica
I20260926 03:34:48.064761 15287 raft_consensus.cc:2229] T 959366c593174546ba571f738626f85a P 1cff45dfdc4c48bc9decf5f844982927 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:34:48.065215 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 14834
I20260926 03:34:48.065382 15287 raft_consensus.cc:2258] T 959366c593174546ba571f738626f85a P 1cff45dfdc4c48bc9decf5f844982927 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:34:48.072793 15287 ts_tablet_manager.cc:1905] T 959366c593174546ba571f738626f85a P 1cff45dfdc4c48bc9decf5f844982927: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:34:48.075214 15289 tablet_replica.cc:330] T 959366c593174546ba571f738626f85a P cb9ebb92eee54c939139cef3b151f9c9: stopping tablet replica
I20260926 03:34:48.075850 15289 raft_consensus.cc:2229] T 959366c593174546ba571f738626f85a P cb9ebb92eee54c939139cef3b151f9c9 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:34:48.076615 15289 raft_consensus.cc:2258] T 959366c593174546ba571f738626f85a P cb9ebb92eee54c939139cef3b151f9c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:34:48.078986 15289 ts_tablet_manager.cc:1905] T 959366c593174546ba571f738626f85a P cb9ebb92eee54c939139cef3b151f9c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:34:48.082051 15288 tablet_replica.cc:330] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d: stopping tablet replica
I20260926 03:34:48.087208 15289 ts_tablet_manager.cc:1918] T 959366c593174546ba571f738626f85a P cb9ebb92eee54c939139cef3b151f9c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:34:48.087427 15289 log.cc:1196] T 959366c593174546ba571f738626f85a P cb9ebb92eee54c939139cef3b151f9c9: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/wal/wals/959366c593174546ba571f738626f85a
I20260926 03:34:48.087971 15288 raft_consensus.cc:2229] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:34:48.088119 15289 ts_tablet_manager.cc:1939] T 959366c593174546ba571f738626f85a P cb9ebb92eee54c939139cef3b151f9c9: Deleting consensus metadata
I20260926 03:34:48.088702 15263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:48.066264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55842 (local address 127.0.97.1:41037)
0926 03:34:48.066555 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:48.066560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:48.066577 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:34:48.066852 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:48.066862 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:48.067021 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:48.067155 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:48.069798 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:48.071272 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:48.073842 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:48.074712 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:48.074834 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:48.079173 (+ 4339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:48.079200 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:48.079226 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:48.079300 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:48.084016 (+ 4716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:48.085103 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:48.085113 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:48.085116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:48.085300 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:48.085759 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:48.085764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:48.085768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:48.086337 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:48.086652 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:48.087992 (+ 1340us) server_negotiation.cc:299] Negotiation successful
0926 03:34:48.088255 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177}
I20260926 03:34:48.089102 15288 raft_consensus.cc:2258] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:34:48.089520 14978 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:34:48.087287 15287 ts_tablet_manager.cc:1918] T 959366c593174546ba571f738626f85a P 1cff45dfdc4c48bc9decf5f844982927: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:34:48.089859 15287 log.cc:1196] T 959366c593174546ba571f738626f85a P 1cff45dfdc4c48bc9decf5f844982927: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/wal/wals/959366c593174546ba571f738626f85a
I20260926 03:34:48.090479 14781 catalog_manager.cc:4841] TS cb9ebb92eee54c939139cef3b151f9c9 (127.0.97.3:45531): tablet 959366c593174546ba571f738626f85a (table test-table [id=5331724f01b146c6a2fd87c92c48406f]) successfully deleted
I20260926 03:34:48.090883 15287 ts_tablet_manager.cc:1939] T 959366c593174546ba571f738626f85a P 1cff45dfdc4c48bc9decf5f844982927: Deleting consensus metadata
I20260926 03:34:48.093271 14782 catalog_manager.cc:4841] TS 1cff45dfdc4c48bc9decf5f844982927 (127.0.97.2:35639): tablet 959366c593174546ba571f738626f85a (table test-table [id=5331724f01b146c6a2fd87c92c48406f]) successfully deleted
I20260926 03:34:48.232713 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 14834
W20260926 03:34:48.254424 14782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:41037 (error 108)
I20260926 03:34:48.254667 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 14987
I20260926 03:34:48.259171 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:48.256402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41037 (local address 127.0.0.1:55854)
0926 03:34:48.258776 (+ 2374us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:48.258919 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2233,"thread_start_us":2131,"threads_started":1}
W20260926 03:34:48.259486 14782 catalog_manager.cc:4584] TS 3511b88ddb2643189dceaea58e35ce9d (127.0.97.1:41037): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 959366c593174546ba571f738626f85a: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41037: connect: Connection refused (error 111)
I20260926 03:34:48.271380 15258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:48.255670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45658 (local address 127.0.97.2:35639)
0926 03:34:48.255878 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:48.255883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:48.255998 (+ 115us) server_negotiation.cc:407] Connection header received
0926 03:34:48.256255 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:48.256265 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:48.256393 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:48.256516 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:48.258452 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:48.259516 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:48.261093 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:48.261518 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:48.261656 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:48.264496 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:48.264512 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:48.264531 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:48.264617 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:48.267965 (+ 3348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:48.269070 (+ 1105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:48.269078 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:48.269080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:48.269212 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:48.269614 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:48.269618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:48.269621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:48.270059 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:48.270401 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:48.270767 (+ 366us) server_negotiation.cc:299] Negotiation successful
0926 03:34:48.271015 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 03:34:48.272387 15110 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:34:48.313607 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:48.312945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41037 (local address 127.0.0.1:55870)
0926 03:34:48.313230 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:48.313389 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260926 03:34:48.315055 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:48.314567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41037 (local address 127.0.0.1:55884)
0926 03:34:48.314776 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:48.314873 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 03:34:48.370765 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 14987
I20260926 03:34:48.376252 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:48.375587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41037 (local address 127.0.0.1:55890)
0926 03:34:48.375859 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:48.375998 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 03:34:48.377668 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:48.377214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41037 (local address 127.0.0.1:55902)
0926 03:34:48.377401 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:48.377499 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 03:34:48.392079 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 15119
I20260926 03:34:48.408479 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:48.393006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37434 (local address 127.0.97.3:45531)
0926 03:34:48.393209 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:48.393216 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:48.393327 (+ 111us) server_negotiation.cc:407] Connection header received
0926 03:34:48.393577 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:48.393583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:48.393668 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:48.393799 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:48.395449 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:48.396335 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:48.397871 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:48.398359 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:48.398539 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:48.401520 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:48.401533 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:48.401558 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:48.401606 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:48.405308 (+ 3702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:48.406086 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:48.406094 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:48.406096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:48.406264 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:48.406671 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:48.406678 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:48.406681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:48.407070 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:48.407446 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:48.407759 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 03:34:48.408004 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 03:34:48.409363 15242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:34:48.458706 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:48.457925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41037 (local address 127.0.0.1:55904)
0926 03:34:48.458313 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:48.458452 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260926 03:34:48.460039 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:48.459525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41037 (local address 127.0.0.1:55916)
0926 03:34:48.459774 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:48.459866 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260926 03:34:48.553387 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 15119
I20260926 03:34:48.576337 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 14764
I20260926 03:34:48.577765 14827 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:34:48.708702 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:48.708004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41037 (local address 127.0.0.1:55928)
0926 03:34:48.708277 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:48.708430 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 03:34:48.708786 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 14764
I20260926 03:34:48.734819 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46679
--webserver_interface=127.0.97.62
--webserver_port=35667
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46679
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:34:49.117122 15296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:34:49.117494 15296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:34:49.117667 15296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:34:49.125761 15296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:34:49.125870 15296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:34:49.125962 15296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:34:49.126073 15296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:34:49.126147 15296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:34:49.136543 15296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=35667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:34:49.138255 15296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:34:49.139883 15296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:34:49.153978 15301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:49.153927 15304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:49.153885 15302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:34:49.154994 15296 server_base.cc:1034] running on GCE node
I20260926 03:34:49.156189 15296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:34:49.158074 15296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:34:49.159323 15296 hybrid_clock.cc:648] HybridClock initialized: now 1790393689159262 us; error 72 us; skew 500 ppm
I20260926 03:34:49.159708 15296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:34:49 dist-test-slave-dt7p krb5kdc[14210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:34:49.163398 15296 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:34:49.165386 15296 webserver.cc:460] Webserver started at http://127.0.97.62:35667/ using document root <none> and password file <none>
I20260926 03:34:49.166203 15296 fs_manager.cc:359] Metadata directory not provided
I20260926 03:34:49.166365 15296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:34:49.172710 15296 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 03:34:49.175890 15311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:49.177510 15296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:34:49.177709 15296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "57587beacbcd425cb35e2e05c483e514"
format_stamp: "Formatted at 2026-09-26 03:34:42 on dist-test-slave-dt7p"
server_key: "024b68f856ef40629c1bb4e9181d454e"
server_key_iv: "a59f70e151481a7f3ee2bb49d7763389"
server_key_version: "encryptionkey@0"
I20260926 03:34:49.178857 15296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:34:49.206107 15296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:34:49.211279 15296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:34:49.211622 15296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:34:49.233083 15296 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46679
I20260926 03:34:49.233107 15362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46679 every 8 connection(s)
I20260926 03:34:49.234915 15296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:34:49.242477 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 15296
I20260926 03:34:49.242839 15363 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:34:49.243449 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:41037
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46679
--webserver_port=35465
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:34:49.246552 15363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514: Bootstrap starting.
I20260926 03:34:49.269979 15363 log.cc:824] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514: Log is configured to *not* fsync() on all Append() calls
I20260926 03:34:49.287556 15363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:34:49.288141 15363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514: Bootstrap complete.
I20260926 03:34:49.298017 15363 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57587beacbcd425cb35e2e05c483e514" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46679 } }
I20260926 03:34:49.299000 15363 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57587beacbcd425cb35e2e05c483e514, State: Initialized, Role: FOLLOWER
I20260926 03:34:49.299729 15363 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57587beacbcd425cb35e2e05c483e514" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46679 } }
I20260926 03:34:49.300000 15363 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:34:49.300148 15363 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:34:49.300341 15363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [term 2 FOLLOWER]: Advancing to term 3
I20260926 03:34:49.305137 15363 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57587beacbcd425cb35e2e05c483e514" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46679 } }
I20260926 03:34:49.305765 15363 leader_election.cc:302] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57587beacbcd425cb35e2e05c483e514; no voters:
I20260926 03:34:49.306489 15363 leader_election.cc:288] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 03:34:49.306720 15367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 03:34:49.307915 15367 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [term 3 LEADER]: Becoming Leader. State: Replica: 57587beacbcd425cb35e2e05c483e514, State: Running, Role: LEADER
I20260926 03:34:49.308808 15363 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:34:49.308630 15367 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57587beacbcd425cb35e2e05c483e514" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46679 } }
I20260926 03:34:49.311697 15369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57587beacbcd425cb35e2e05c483e514. Latest consensus state: current_term: 3 leader_uuid: "57587beacbcd425cb35e2e05c483e514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57587beacbcd425cb35e2e05c483e514" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46679 } } }
I20260926 03:34:49.312084 15369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [sys.catalog]: This master's current role is: LEADER
I20260926 03:34:49.312716 15368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "57587beacbcd425cb35e2e05c483e514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57587beacbcd425cb35e2e05c483e514" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46679 } } }
I20260926 03:34:49.313005 15368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57587beacbcd425cb35e2e05c483e514 [sys.catalog]: This master's current role is: LEADER
I20260926 03:34:49.318869 15374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:34:49.328408 15374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:34:49.330003 15374 catalog_manager.cc:1259] Loaded cluster ID: 04964e8c9e844d89b322d8c905759b1d
I20260926 03:34:49.330111 15374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 03:34:49.332267 15374 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f98844b09ad google::LogMessage::Fail() at ??:0
@ 0x7f98844b388a google::LogMessage::SendToLog() at ??:0
@ 0x7f98844b04de google::LogMessage::Flush() at ??:0
@ 0x7f98844b1a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9899ee3512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f9899ee0976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9899f68610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f9899f68439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f98870fc345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f98870ff91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f98870ff799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f98870daeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f988a3b46db start_thread at ??:0
@ 0x7f988334a61f clone at ??:0
W20260926 03:34:49.696836 15365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:34:49.697144 15365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:34:49.697319 15365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:34:49.704993 15365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:34:49.705154 15365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:34:49.705359 15365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:34:49.716012 15365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:41037
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=35465
--tserver_master_addrs=127.0.97.62:46679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:34:49.717710 15365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:34:49.719571 15365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:34:49.734243 15405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:49.734247 15406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:49.734318 15408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:34:49.735077 15365 server_base.cc:1034] running on GCE node
I20260926 03:34:49.735743 15365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:34:49.736786 15365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:34:49.737990 15365 hybrid_clock.cc:648] HybridClock initialized: now 1790393689737933 us; error 47 us; skew 500 ppm
I20260926 03:34:49.738355 15365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:34:49 dist-test-slave-dt7p krb5kdc[14210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:34:49.741958 15365 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:34:49.743521 15365 webserver.cc:460] Webserver started at http://127.0.97.1:35465/ using document root <none> and password file <none>
I20260926 03:34:49.744287 15365 fs_manager.cc:359] Metadata directory not provided
I20260926 03:34:49.744426 15365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:34:49.750244 15365 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 03:34:49.753381 15415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:49.754846 15365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:34:49.755034 15365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/wal
uuid: "3511b88ddb2643189dceaea58e35ce9d"
format_stamp: "Formatted at 2026-09-26 03:34:43 on dist-test-slave-dt7p"
server_key: "cc8b2b244a878c26f943c89be916d847"
server_key_iv: "1301223c8b27074a4b02588290d98e64"
server_key_version: "encryptionkey@0"
I20260926 03:34:49.756147 15365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:34:49.771080 15365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:34:49.775738 15365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:34:49.776031 15365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:34:49.777102 15365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:34:49.779042 15422 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260926 03:34:49.785303 15365 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 03:34:49.785480 15365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260926 03:34:49.785624 15365 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 03:34:49.789031 15365 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 03:34:49.789137 15365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.003s sys 0.000s
I20260926 03:34:49.789482 15422 tablet_bootstrap.cc:492] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d: Bootstrap starting.
I20260926 03:34:49.798578 15422 log.cc:824] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d: Log is configured to *not* fsync() on all Append() calls
I20260926 03:34:49.814939 15422 tablet_bootstrap.cc:492] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:34:49.815678 15422 tablet_bootstrap.cc:492] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d: Bootstrap complete.
I20260926 03:34:49.816244 15422 ts_tablet_manager.cc:1397] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d: Time spent bootstrapping tablet: real 0.027s user 0.019s sys 0.004s
I20260926 03:34:49.826987 15422 raft_consensus.cc:348] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff45dfdc4c48bc9decf5f844982927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35639 } } peers { permanent_uuid: "3511b88ddb2643189dceaea58e35ce9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41037 } } peers { permanent_uuid: "cb9ebb92eee54c939139cef3b151f9c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45531 } }
I20260926 03:34:49.828567 15422 raft_consensus.cc:729] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3511b88ddb2643189dceaea58e35ce9d, State: Initialized, Role: FOLLOWER
I20260926 03:34:49.829669 15422 consensus_queue.cc:260] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff45dfdc4c48bc9decf5f844982927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35639 } } peers { permanent_uuid: "3511b88ddb2643189dceaea58e35ce9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41037 } } peers { permanent_uuid: "cb9ebb92eee54c939139cef3b151f9c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45531 } }
I20260926 03:34:49.831709 15422 ts_tablet_manager.cc:1428] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d: Time spent starting tablet: real 0.015s user 0.017s sys 0.000s
I20260926 03:34:49.839562 15365 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41037
I20260926 03:34:49.839598 15530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41037 every 8 connection(s)
I20260926 03:34:49.841208 15365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:34:49.844774 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 15365
I20260926 03:34:49.845690 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:35639
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46679
--webserver_port=40145
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:34:49.847059 15533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:49.844340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.1:35265)
0926 03:34:49.845482 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:49.845741 (+ 259us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":775,"thread_start_us":519,"threads_started":1}
W20260926 03:34:49.847872 15531 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46679 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:46679: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
I20260926 03:34:49.848917 15533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:49.848451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.1:56537)
0926 03:34:49.848661 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:49.848755 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 03:34:49.849886 15533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:49.849446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.1:53359)
0926 03:34:49.849616 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:49.849714 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 03:34:49.850140 15531 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 03:34:50.231426 15534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:34:50.231775 15534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:34:50.231956 15534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:34:50.239902 15534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:34:50.240041 15534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:34:50.240262 15534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:34:50.251013 15534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:35639
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=40145
--tserver_master_addrs=127.0.97.62:46679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:34:50.252719 15534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:34:50.254311 15534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:34:50.269665 15540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:50.269708 15542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:50.269732 15539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:34:50.270705 15534 server_base.cc:1034] running on GCE node
I20260926 03:34:50.271370 15534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:34:50.272416 15534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:34:50.273667 15534 hybrid_clock.cc:648] HybridClock initialized: now 1790393690273597 us; error 65 us; skew 500 ppm
I20260926 03:34:50.274017 15534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:34:50 dist-test-slave-dt7p krb5kdc[14210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:34:50.282413 15534 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:34:50.284123 15534 webserver.cc:460] Webserver started at http://127.0.97.2:40145/ using document root <none> and password file <none>
I20260926 03:34:50.285017 15534 fs_manager.cc:359] Metadata directory not provided
I20260926 03:34:50.285171 15534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:34:50.291452 15534 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260926 03:34:50.294703 15549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:50.296185 15534 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:34:50.296387 15534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/wal
uuid: "1cff45dfdc4c48bc9decf5f844982927"
format_stamp: "Formatted at 2026-09-26 03:34:43 on dist-test-slave-dt7p"
server_key: "8a1a4c57ef90077cdce1ecdd3e73f827"
server_key_iv: "5afae4554131e9a0c80d726fa118248e"
server_key_version: "encryptionkey@0"
I20260926 03:34:50.297622 15534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:34:50.325881 15534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:34:50.331207 15534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:34:50.331553 15534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:34:50.332810 15534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:34:50.334373 15534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:34:50.334501 15534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:50.334642 15534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:34:50.334734 15534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:50.373932 15534 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35639
I20260926 03:34:50.373991 15661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35639 every 8 connection(s)
I20260926 03:34:50.375659 15534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:34:50.381222 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:50.379100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.2:45617)
0926 03:34:50.379915 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:50.380167 (+ 252us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":321,"threads_started":1}
W20260926 03:34:50.381917 15662 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46679 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:46679: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
I20260926 03:34:50.382766 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:50.382435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.2:38605)
0926 03:34:50.382580 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:50.382646 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 03:34:50.383574 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:50.383245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.2:34855)
0926 03:34:50.383381 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:50.383459 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 03:34:50.383800 15662 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 03:34:50.384979 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 15534
I20260926 03:34:50.385669 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:45531
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46679
--webserver_port=38903
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:34:50.763536 15665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:34:50.763881 15665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:34:50.763988 15665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:34:50.771975 15665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:34:50.772113 15665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:34:50.772251 15665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:34:50.783053 15665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:45531
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=38903
--tserver_master_addrs=127.0.97.62:46679
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:34:50.784622 15665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:34:50.786327 15665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:34:50.801192 15671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:50.801317 15673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:34:50.801517 15670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:34:50.802891 15665 server_base.cc:1034] running on GCE node
I20260926 03:34:50.803501 15665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:34:50.804476 15665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:34:50.805676 15665 hybrid_clock.cc:648] HybridClock initialized: now 1790393690805615 us; error 58 us; skew 500 ppm
I20260926 03:34:50.805984 15665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:34:50 dist-test-slave-dt7p krb5kdc[14210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:34:50.809456 15665 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:34:50.811115 15665 webserver.cc:460] Webserver started at http://127.0.97.3:38903/ using document root <none> and password file <none>
I20260926 03:34:50.811925 15665 fs_manager.cc:359] Metadata directory not provided
I20260926 03:34:50.812075 15665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:34:50.818181 15665 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 03:34:50.821316 15680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:50.822975 15665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:34:50.823158 15665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/wal
uuid: "cb9ebb92eee54c939139cef3b151f9c9"
format_stamp: "Formatted at 2026-09-26 03:34:44 on dist-test-slave-dt7p"
server_key: "6148b146126b8f37a04fe7a627f95e41"
server_key_iv: "bae1a42b5a61105223a88eeeb1c898fa"
server_key_version: "encryptionkey@0"
I20260926 03:34:50.824281 15665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:34:50.841620 15665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:34:50.846886 15665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:34:50.847216 15665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:34:50.848326 15665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:34:50.849908 15665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:34:50.850047 15665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:50.850179 15665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:34:50.850283 15665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:34:50.851753 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:50.850833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.1:42641)
0926 03:34:50.851401 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:50.851499 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":271,"threads_started":1}
I20260926 03:34:50.888123 15665 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45531
I20260926 03:34:50.888180 15793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45531 every 8 connection(s)
I20260926 03:34:50.890004 15665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:34:50.892946 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 15665
I20260926 03:34:50.896148 15796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:50.893553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.3:34693)
0926 03:34:50.894645 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:50.894949 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":378,"threads_started":1}
W20260926 03:34:50.896944 15794 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46679 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:46679: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
I20260926 03:34:50.897861 15796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:50.897510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.3:41751)
0926 03:34:50.897659 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:50.897741 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 03:34:50.898818 15796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:50.898501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.3:45443)
0926 03:34:50.898625 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:50.898697 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 03:34:50.899071 15794 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 03:34:51.206734 15798 raft_consensus.cc:482] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:34:51.207122 15798 raft_consensus.cc:504] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff45dfdc4c48bc9decf5f844982927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35639 } } peers { permanent_uuid: "3511b88ddb2643189dceaea58e35ce9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41037 } } peers { permanent_uuid: "cb9ebb92eee54c939139cef3b151f9c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45531 } }
I20260926 03:34:51.209470 15798 leader_election.cc:288] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cff45dfdc4c48bc9decf5f844982927 (127.0.97.2:35639), cb9ebb92eee54c939139cef3b151f9c9 (127.0.97.3:45531)
Sep 26 03:34:51 dist-test-slave-dt7p krb5kdc[14210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
Sep 26 03:34:51 dist-test-slave-dt7p krb5kdc[14210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:34:51.234921 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:51.209766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35639 (local address 127.0.97.1:34265)
0926 03:34:51.210093 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:51.210149 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 03:34:51.210924 (+ 775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:34:51.216010 (+ 5086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:34:51.216018 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:34:51.216816 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:34:51.217579 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:51.217608 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:51.219232 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:51.219244 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:51.220801 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:51.220814 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:51.221953 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:51.221960 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:51.222277 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:51.223237 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:34:51.223275 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:34:51.226114 (+ 2839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:34:51.231026 (+ 4912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:51.231034 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:51.231047 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:51.231513 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:51.231997 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:51.232004 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:51.232008 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:51.232199 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:51.233167 (+ 968us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:34:51.233176 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:34:51.233533 (+ 357us) client_negotiation.cc:769] Sending connection context
0926 03:34:51.233835 (+ 302us) client_negotiation.cc:240] Negotiation successful
0926 03:34:51.234459 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":112,"spinlock_wait_cycles":15232}
I20260926 03:34:51.235347 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:51.210587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47995 (local address 127.0.97.3:45531)
0926 03:34:51.212395 (+ 1808us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:51.212409 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:51.212449 (+ 40us) server_negotiation.cc:407] Connection header received
0926 03:34:51.214842 (+ 2393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:51.214856 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:51.215445 (+ 589us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:51.215855 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:51.217783 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:51.219037 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:51.220962 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:51.221695 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:51.221949 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:51.226707 (+ 4758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:51.226713 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:51.226724 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:51.226764 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:51.230862 (+ 4098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:51.232111 (+ 1249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:51.232117 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:51.232121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:51.232281 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:51.232697 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:51.232702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:51.232704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:51.233192 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:51.233421 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:51.234503 (+ 1082us) server_negotiation.cc:299] Negotiation successful
0926 03:34:51.234899 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1180,"thread_start_us":1001,"threads_started":1}
I20260926 03:34:51.235363 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:51.210330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34265 (local address 127.0.97.2:35639)
0926 03:34:51.212300 (+ 1970us) server_negotiation.cc:206] Beginning negotiation
0926 03:34:51.212315 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 03:34:51.212362 (+ 47us) server_negotiation.cc:407] Connection header received
0926 03:34:51.214848 (+ 2486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:34:51.214858 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:34:51.215441 (+ 583us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:34:51.215855 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:34:51.217792 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:51.219036 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:51.220952 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:34:51.221656 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:34:51.221953 (+ 297us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:51.226479 (+ 4526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:34:51.226494 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:34:51.226518 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:34:51.226576 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:34:51.230872 (+ 4296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:51.231670 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:51.231677 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:51.231683 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:51.231870 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:34:51.232348 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:34:51.232354 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:34:51.232357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:34:51.233029 (+ 672us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:34:51.233242 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 03:34:51.234080 (+ 838us) server_negotiation.cc:299] Negotiation successful
0926 03:34:51.234924 (+ 844us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1769,"thread_start_us":1635,"threads_started":1}
I20260926 03:34:51.235924 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:51.209799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45531 (local address 127.0.97.1:47995)
0926 03:34:51.210709 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:51.210738 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 03:34:51.210947 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:34:51.216516 (+ 5569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:34:51.216519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:34:51.217150 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:34:51.217578 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:51.217601 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:51.219243 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:51.219247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:51.220813 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:34:51.220825 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:34:51.221977 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:34:51.221983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:34:51.222281 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:34:51.223604 (+ 1323us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:34:51.223628 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:34:51.226443 (+ 2815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:34:51.231629 (+ 5186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:51.231632 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:51.231634 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:51.231947 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:51.232398 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:34:51.232406 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:34:51.232409 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:34:51.232547 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:34:51.233335 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:34:51.233339 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:34:51.233532 (+ 193us) client_negotiation.cc:769] Sending connection context
0926 03:34:51.233828 (+ 296us) client_negotiation.cc:240] Negotiation successful
0926 03:34:51.235547 (+ 1719us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":1371,"spinlock_wait_cycles":19328,"thread_start_us":1034,"threads_started":1}
W20260926 03:34:51.238914 15417 leader_election.cc:341] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1cff45dfdc4c48bc9decf5f844982927 (127.0.97.2:35639): Not found: Tablet not found: 959366c593174546ba571f738626f85a
W20260926 03:34:51.239167 15416 leader_election.cc:341] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cb9ebb92eee54c939139cef3b151f9c9 (127.0.97.3:45531): Not found: Tablet not found: 959366c593174546ba571f738626f85a
I20260926 03:34:51.239347 15416 leader_election.cc:302] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3511b88ddb2643189dceaea58e35ce9d; no voters: 1cff45dfdc4c48bc9decf5f844982927, cb9ebb92eee54c939139cef3b151f9c9
I20260926 03:34:51.239786 15798 raft_consensus.cc:2726] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 03:34:51.385419 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:51.384425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.2:42353)
0926 03:34:51.385006 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:51.385140 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":297,"threads_started":1}
I20260926 03:34:51.853673 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:51.852853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.1:40873)
0926 03:34:51.853364 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:51.853462 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":240,"threads_started":1}
I20260926 03:34:51.901008 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:51.899866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.3:59209)
0926 03:34:51.900494 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:51.900694 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":326,"threads_started":1}
I20260926 03:34:52.387290 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:52.386434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.2:44579)
0926 03:34:52.386980 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:52.387069 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":326,"threads_started":1}
I20260926 03:34:52.855865 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:52.854839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.1:41341)
0926 03:34:52.855472 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:52.855608 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":334,"threads_started":1}
I20260926 03:34:52.903414 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:52.902406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.3:49809)
0926 03:34:52.903049 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:52.903160 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":343,"threads_started":1}
I20260926 03:34:53.316751 15808 raft_consensus.cc:482] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:34:53.317021 15808 raft_consensus.cc:504] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff45dfdc4c48bc9decf5f844982927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35639 } } peers { permanent_uuid: "3511b88ddb2643189dceaea58e35ce9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41037 } } peers { permanent_uuid: "cb9ebb92eee54c939139cef3b151f9c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45531 } }
I20260926 03:34:53.317795 15808 leader_election.cc:288] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cff45dfdc4c48bc9decf5f844982927 (127.0.97.2:35639), cb9ebb92eee54c939139cef3b151f9c9 (127.0.97.3:45531)
W20260926 03:34:53.319052 15417 leader_election.cc:341] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1cff45dfdc4c48bc9decf5f844982927 (127.0.97.2:35639): Not found: Tablet not found: 959366c593174546ba571f738626f85a
W20260926 03:34:53.319378 15416 leader_election.cc:341] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cb9ebb92eee54c939139cef3b151f9c9 (127.0.97.3:45531): Not found: Tablet not found: 959366c593174546ba571f738626f85a
I20260926 03:34:53.319530 15416 leader_election.cc:302] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3511b88ddb2643189dceaea58e35ce9d; no voters: 1cff45dfdc4c48bc9decf5f844982927, cb9ebb92eee54c939139cef3b151f9c9
I20260926 03:34:53.319809 15808 raft_consensus.cc:2726] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 03:34:53.389343 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:53.388400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.2:43535)
0926 03:34:53.389054 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:53.389143 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":351,"threads_started":1}
I20260926 03:34:53.857995 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:53.857175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.1:39667)
0926 03:34:53.857692 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:53.857790 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":269,"threads_started":1}
I20260926 03:34:53.905716 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:53.904526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.3:43253)
0926 03:34:53.905344 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:53.905455 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":523,"threads_started":1}
I20260926 03:34:54.391460 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:54.390503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.2:54315)
0926 03:34:54.391061 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:54.391185 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":262,"threads_started":1}
I20260926 03:34:54.860296 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:54.859190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.1:35083)
0926 03:34:54.859967 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:54.860062 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":432,"threads_started":1}
I20260926 03:34:54.907653 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:54.906772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.3:45045)
0926 03:34:54.907320 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:54.907430 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":270,"threads_started":1}
I20260926 03:34:55.079056 15815 raft_consensus.cc:482] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:34:55.079325 15815 raft_consensus.cc:504] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff45dfdc4c48bc9decf5f844982927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35639 } } peers { permanent_uuid: "3511b88ddb2643189dceaea58e35ce9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41037 } } peers { permanent_uuid: "cb9ebb92eee54c939139cef3b151f9c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45531 } }
I20260926 03:34:55.080190 15815 leader_election.cc:288] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cff45dfdc4c48bc9decf5f844982927 (127.0.97.2:35639), cb9ebb92eee54c939139cef3b151f9c9 (127.0.97.3:45531)
W20260926 03:34:55.081418 15417 leader_election.cc:341] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1cff45dfdc4c48bc9decf5f844982927 (127.0.97.2:35639): Not found: Tablet not found: 959366c593174546ba571f738626f85a
W20260926 03:34:55.081712 15416 leader_election.cc:341] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cb9ebb92eee54c939139cef3b151f9c9 (127.0.97.3:45531): Not found: Tablet not found: 959366c593174546ba571f738626f85a
I20260926 03:34:55.081861 15416 leader_election.cc:302] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3511b88ddb2643189dceaea58e35ce9d; no voters: 1cff45dfdc4c48bc9decf5f844982927, cb9ebb92eee54c939139cef3b151f9c9
I20260926 03:34:55.082163 15815 raft_consensus.cc:2726] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 03:34:55.394012 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:55.392956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.2:35709)
0926 03:34:55.393659 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:55.393756 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":407,"threads_started":1}
I20260926 03:34:55.862227 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:55.861373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.1:47815)
0926 03:34:55.861919 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:55.862020 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":281,"threads_started":1}
I20260926 03:34:55.909885 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:55.908808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.3:49823)
0926 03:34:55.909497 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:55.909636 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":370,"threads_started":1}
I20260926 03:34:56.395987 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:56.395079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.2:34687)
0926 03:34:56.395648 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:56.395762 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":306,"threads_started":1}
I20260926 03:34:56.864106 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:56.863303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.1:41047)
0926 03:34:56.863776 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:56.863865 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":263,"threads_started":1}
I20260926 03:34:56.911698 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:56.910862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.3:42589)
0926 03:34:56.911387 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:56.911477 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":302,"threads_started":1}
I20260926 03:34:57.398258 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:57.397223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.2:56867)
0926 03:34:57.397878 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:57.397985 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":277,"threads_started":1}
I20260926 03:34:57.866241 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:57.865301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.1:33809)
0926 03:34:57.865919 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:57.866039 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":285,"threads_started":1}
I20260926 03:34:57.914125 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:57.912728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.3:39443)
0926 03:34:57.913511 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:57.913675 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":330,"threads_started":1}
I20260926 03:34:57.992100 15825 raft_consensus.cc:482] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:34:57.992396 15825 raft_consensus.cc:504] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff45dfdc4c48bc9decf5f844982927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35639 } } peers { permanent_uuid: "3511b88ddb2643189dceaea58e35ce9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41037 } } peers { permanent_uuid: "cb9ebb92eee54c939139cef3b151f9c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45531 } }
I20260926 03:34:57.993269 15825 leader_election.cc:288] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cff45dfdc4c48bc9decf5f844982927 (127.0.97.2:35639), cb9ebb92eee54c939139cef3b151f9c9 (127.0.97.3:45531)
W20260926 03:34:57.994436 15416 leader_election.cc:341] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cb9ebb92eee54c939139cef3b151f9c9 (127.0.97.3:45531): Not found: Tablet not found: 959366c593174546ba571f738626f85a
W20260926 03:34:57.994603 15417 leader_election.cc:341] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1cff45dfdc4c48bc9decf5f844982927 (127.0.97.2:35639): Not found: Tablet not found: 959366c593174546ba571f738626f85a
I20260926 03:34:57.994779 15417 leader_election.cc:302] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3511b88ddb2643189dceaea58e35ce9d; no voters: 1cff45dfdc4c48bc9decf5f844982927, cb9ebb92eee54c939139cef3b151f9c9
I20260926 03:34:57.995060 15825 raft_consensus.cc:2726] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 03:34:58.400199 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:58.399369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.2:38511)
0926 03:34:58.399879 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:58.399971 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":263,"threads_started":1}
I20260926 03:34:58.868263 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:58.867268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.1:41851)
0926 03:34:58.867916 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:58.868020 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":361,"threads_started":1}
I20260926 03:34:58.915956 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:58.915180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.3:45245)
0926 03:34:58.915643 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:58.915733 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":241,"threads_started":1}
I20260926 03:34:59.402163 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:59.401248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.2:40457)
0926 03:34:59.401868 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:59.401958 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":342,"threads_started":1}
I20260926 03:34:59.870383 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:59.869498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.1:44603)
0926 03:34:59.870041 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:59.870140 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":290,"threads_started":1}
I20260926 03:34:59.918196 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:34:59.917099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.3:54363)
0926 03:34:59.917756 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0926 03:34:59.917974 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":331,"threads_started":1}
I20260926 03:35:00.404235 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:00.403293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.2:37027)
0926 03:35:00.403897 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:00.403993 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":345,"threads_started":1}
I20260926 03:35:00.872458 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:00.871519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.1:52839)
0926 03:35:00.872097 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:00.872241 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":325,"threads_started":1}
I20260926 03:35:00.920156 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:00.919333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.3:44915)
0926 03:35:00.919811 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:00.919900 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":259,"threads_started":1}
I20260926 03:35:01.406177 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:01.405333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.2:60781)
0926 03:35:01.405871 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:01.405972 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":251,"threads_started":1}
I20260926 03:35:01.874435 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:01.873562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.1:55577)
0926 03:35:01.874091 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:01.874218 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":275,"threads_started":1}
I20260926 03:35:01.922199 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:01.921244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.3:42553)
0926 03:35:01.921890 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:01.921993 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":296,"threads_started":1}
I20260926 03:35:02.408061 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:02.407226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.2:52577)
0926 03:35:02.407747 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:02.407840 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":259,"threads_started":1}
I20260926 03:35:02.876443 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:02.875515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.1:54141)
0926 03:35:02.876074 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:02.876214 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":287,"threads_started":1}
I20260926 03:35:02.924052 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:02.923211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.3:48891)
0926 03:35:02.923730 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:02.923820 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":275,"threads_started":1}
I20260926 03:35:03.410281 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:03.409276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.2:37749)
0926 03:35:03.409950 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:03.410053 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":363,"threads_started":1}
I20260926 03:35:03.878517 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:03.877619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.1:57785)
0926 03:35:03.878156 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:03.878262 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":274,"threads_started":1}
I20260926 03:35:03.926048 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:03.925252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.3:38779)
0926 03:35:03.925761 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:03.925853 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":250,"threads_started":1}
I20260926 03:35:04.412225 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:04.411314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.2:45027)
0926 03:35:04.411870 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:04.411965 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":281,"threads_started":1}
I20260926 03:35:04.881058 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:04.880005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.1:44571)
0926 03:35:04.880678 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:04.880777 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":321,"threads_started":1}
I20260926 03:35:04.927937 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:04.927025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.3:38469)
0926 03:35:04.927607 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:04.927699 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":338,"threads_started":1}
I20260926 03:35:05.289548 15847 raft_consensus.cc:482] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:35:05.289825 15847 raft_consensus.cc:504] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cff45dfdc4c48bc9decf5f844982927" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35639 } } peers { permanent_uuid: "3511b88ddb2643189dceaea58e35ce9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41037 } } peers { permanent_uuid: "cb9ebb92eee54c939139cef3b151f9c9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45531 } }
I20260926 03:35:05.290642 15847 leader_election.cc:288] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cff45dfdc4c48bc9decf5f844982927 (127.0.97.2:35639), cb9ebb92eee54c939139cef3b151f9c9 (127.0.97.3:45531)
W20260926 03:35:05.292076 15416 leader_election.cc:341] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cb9ebb92eee54c939139cef3b151f9c9 (127.0.97.3:45531): Not found: Tablet not found: 959366c593174546ba571f738626f85a
W20260926 03:35:05.292336 15417 leader_election.cc:341] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1cff45dfdc4c48bc9decf5f844982927 (127.0.97.2:35639): Not found: Tablet not found: 959366c593174546ba571f738626f85a
I20260926 03:35:05.292516 15417 leader_election.cc:302] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3511b88ddb2643189dceaea58e35ce9d; no voters: 1cff45dfdc4c48bc9decf5f844982927, cb9ebb92eee54c939139cef3b151f9c9
I20260926 03:35:05.292905 15847 raft_consensus.cc:2726] T 959366c593174546ba571f738626f85a P 3511b88ddb2643189dceaea58e35ce9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 03:35:05.414644 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:05.413638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.2:33085)
0926 03:35:05.414264 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:05.414392 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":309,"threads_started":1}
I20260926 03:35:05.883042 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:05.882220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.1:47981)
0926 03:35:05.882725 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:05.882829 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":268,"threads_started":1}
I20260926 03:35:05.901281 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 15365
I20260926 03:35:05.922614 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:05.902753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53472 (local address 127.0.97.1:41037)
0926 03:35:05.904873 (+ 2120us) server_negotiation.cc:206] Beginning negotiation
0926 03:35:05.904887 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:35:05.904927 (+ 40us) server_negotiation.cc:407] Connection header received
0926 03:35:05.905022 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:35:05.905032 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:35:05.905413 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:35:05.905900 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:35:05.907930 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:05.909693 (+ 1763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:05.911720 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:05.912621 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:05.912773 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:05.916050 (+ 3277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:35:05.916065 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:35:05.916085 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:35:05.916135 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:35:05.919335 (+ 3200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:35:05.920119 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:35:05.920128 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:35:05.920130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:35:05.920287 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:35:05.920730 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:35:05.920737 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:35:05.920740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:35:05.921174 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:35:05.921518 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0926 03:35:05.921880 (+ 362us) server_negotiation.cc:299] Negotiation successful
0926 03:35:05.922209 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1781,"thread_start_us":1522,"threads_started":1}
I20260926 03:35:05.924122 15526 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:35:05.930290 15852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:05.929360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46679 (local address 127.0.97.3:45747)
0926 03:35:05.929954 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:05.930068 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":305,"threads_started":1}
I20260926 03:35:06.026628 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 15365
I20260926 03:35:06.047595 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 15534
I20260926 03:35:06.062631 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:06.048652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47510 (local address 127.0.97.2:35639)
0926 03:35:06.049255 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0926 03:35:06.049260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:35:06.049278 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:35:06.049349 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:35:06.049354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:35:06.049425 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:35:06.049530 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:35:06.051009 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:06.051912 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:06.053419 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:06.053919 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:06.054038 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:06.057066 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:35:06.057071 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:35:06.057074 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:35:06.057116 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:35:06.060013 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:35:06.060794 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:35:06.060798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:35:06.060801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:35:06.060937 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:35:06.061256 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:35:06.061259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:35:06.061261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:35:06.061569 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:35:06.061808 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 03:35:06.061981 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 03:35:06.062173 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":354,"threads_started":1}
I20260926 03:35:06.063707 15657 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:35:06.156677 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 15534
I20260926 03:35:06.177589 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 15665
I20260926 03:35:06.193260 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:06.178663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60376 (local address 127.0.97.3:45531)
0926 03:35:06.179330 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0926 03:35:06.179335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:35:06.179355 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:35:06.179423 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:35:06.179427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:35:06.179513 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:35:06.179627 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:35:06.181161 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:06.182098 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:06.183443 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:06.183924 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:06.184039 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:06.187006 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:35:06.187012 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:35:06.187016 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:35:06.187061 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:35:06.190266 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:35:06.191046 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:35:06.191051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:35:06.191053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:35:06.191251 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:35:06.191641 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:35:06.191647 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:35:06.191649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:35:06.191970 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:35:06.192255 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0926 03:35:06.192659 (+ 404us) server_negotiation.cc:299] Negotiation successful
0926 03:35:06.192876 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":376,"threads_started":1}
I20260926 03:35:06.194101 15789 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:35:06.286899 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 15665
2026-09-26T03:35:06Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25700 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:35:06 dist-test-slave-dt7p krb5kdc[15859](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:35:06 dist-test-slave-dt7p krb5kdc[15859](info): set up 2 sockets
Sep 26 03:35:06 dist-test-slave-dt7p krb5kdc[15859](info): commencing operation
krb5kdc: starting...
W20260926 03:35:07.417550 388 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:35:07 dist-test-slave-dt7p krb5kdc[15859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:35:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:35:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:35:07.698632 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43539
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:35:08.093339 15872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:35:08.093755 15872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:35:08.093931 15872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:35:08.101703 15872 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 03:35:08.101850 15872 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 03:35:08.101946 15872 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 03:35:08.102051 15872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:35:08.102164 15872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:35:08.102284 15872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:35:08.102367 15872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:35:08.102454 15872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:35:08.113349 15872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:35:08.114945 15872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:35:08.116647 15872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:35:08.130991 15878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:35:08.131052 15880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:35:08.131458 15872 server_base.cc:1034] running on GCE node
W20260926 03:35:08.131102 15877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:35:08.132815 15872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:35:08.135011 15872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:35:08.136495 15872 hybrid_clock.cc:648] HybridClock initialized: now 1790393708136410 us; error 62 us; skew 500 ppm
I20260926 03:35:08.137028 15872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:35:08 dist-test-slave-dt7p krb5kdc[15859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393708, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:35:08.141142 15872 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:35:08.143055 15872 webserver.cc:460] Webserver started at http://127.0.97.62:42683/ using document root <none> and password file <none>
I20260926 03:35:08.143989 15872 fs_manager.cc:359] Metadata directory not provided
I20260926 03:35:08.144140 15872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:35:08.144527 15872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:35:08.148212 15872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "5e2037cbf110405d90315bdb3e5d87c4"
format_stamp: "Formatted at 2026-09-26 03:35:08 on dist-test-slave-dt7p"
server_key: "dd02b3f03e7d752593de335f20be849a"
server_key_iv: "c7fb82582765c1f9b337292ae70e5c36"
server_key_version: "encryptionkey@0"
I20260926 03:35:08.149246 15872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "5e2037cbf110405d90315bdb3e5d87c4"
format_stamp: "Formatted at 2026-09-26 03:35:08 on dist-test-slave-dt7p"
server_key: "dd02b3f03e7d752593de335f20be849a"
server_key_iv: "c7fb82582765c1f9b337292ae70e5c36"
server_key_version: "encryptionkey@0"
I20260926 03:35:08.155694 15872 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260926 03:35:08.160166 15887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:35:08.161787 15872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:35:08.161993 15872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "5e2037cbf110405d90315bdb3e5d87c4"
format_stamp: "Formatted at 2026-09-26 03:35:08 on dist-test-slave-dt7p"
server_key: "dd02b3f03e7d752593de335f20be849a"
server_key_iv: "c7fb82582765c1f9b337292ae70e5c36"
server_key_version: "encryptionkey@0"
I20260926 03:35:08.162237 15872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:35:08.179016 15872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:35:08.183451 15872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:35:08.183753 15872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:35:08.204814 15872 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43539
I20260926 03:35:08.204849 15938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43539 every 8 connection(s)
I20260926 03:35:08.206528 15872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:35:08.208294 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 15872
I20260926 03:35:08.208602 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:35:08.209658 388 external_mini_cluster.cc:1428] Setting key f72899da14575f0fb9f419750a94aeb0
I20260926 03:35:08.212636 15939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 03:35:08 dist-test-slave-dt7p krb5kdc[15859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:35:08.228781 15939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4: Bootstrap starting.
I20260926 03:35:08.234176 15939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4: Neither blocks nor log segments found. Creating new log.
I20260926 03:35:08.235759 15939 log.cc:824] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4: Log is configured to *not* fsync() on all Append() calls
I20260926 03:35:08.238909 15942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:08.212244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60356 (local address 127.0.97.62:43539)
0926 03:35:08.213224 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0926 03:35:08.213238 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:35:08.213291 (+ 53us) server_negotiation.cc:407] Connection header received
0926 03:35:08.215180 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:35:08.215205 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:35:08.215672 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:35:08.216066 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:35:08.218463 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:08.219990 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:08.222353 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:08.223098 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:08.223244 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:08.228886 (+ 5642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:35:08.228901 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:35:08.228925 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:35:08.228992 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:35:08.233559 (+ 4567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:35:08.234496 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:35:08.234505 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:35:08.234513 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:35:08.234717 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:35:08.235254 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:35:08.235258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:35:08.235260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:35:08.235949 (+ 689us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:35:08.236232 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0926 03:35:08.236948 (+ 716us) server_negotiation.cc:299] Negotiation successful
0926 03:35:08.237535 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":465,"threads_started":1}
I20260926 03:35:08.239894 15939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4: No bootstrap required, opened a new log
I20260926 03:35:08.249126 15939 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e2037cbf110405d90315bdb3e5d87c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43539 } }
I20260926 03:35:08.249538 15939 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:35:08.249637 15939 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e2037cbf110405d90315bdb3e5d87c4, State: Initialized, Role: FOLLOWER
I20260926 03:35:08.250238 15939 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e2037cbf110405d90315bdb3e5d87c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43539 } }
I20260926 03:35:08.250469 15939 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:35:08.250591 15939 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:35:08.250751 15939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:35:08.253507 15939 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e2037cbf110405d90315bdb3e5d87c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43539 } }
I20260926 03:35:08.254045 15939 leader_election.cc:302] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e2037cbf110405d90315bdb3e5d87c4; no voters:
I20260926 03:35:08.254894 15939 leader_election.cc:288] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:35:08.255028 15944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:35:08.255884 15944 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [term 1 LEADER]: Becoming Leader. State: Replica: 5e2037cbf110405d90315bdb3e5d87c4, State: Running, Role: LEADER
I20260926 03:35:08.256522 15944 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e2037cbf110405d90315bdb3e5d87c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43539 } }
I20260926 03:35:08.257462 15939 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:35:08.259689 15946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e2037cbf110405d90315bdb3e5d87c4. Latest consensus state: current_term: 1 leader_uuid: "5e2037cbf110405d90315bdb3e5d87c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e2037cbf110405d90315bdb3e5d87c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43539 } } }
I20260926 03:35:08.260154 15946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [sys.catalog]: This master's current role is: LEADER
I20260926 03:35:08.260465 15945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e2037cbf110405d90315bdb3e5d87c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e2037cbf110405d90315bdb3e5d87c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43539 } } }
I20260926 03:35:08.260841 15945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [sys.catalog]: This master's current role is: LEADER
I20260926 03:35:08.265295 15949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:35:08.271555 15949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:35:08.282097 15949 catalog_manager.cc:1347] Generated new cluster ID: 7d88b04aedf44fffa5e35a9554ef5ec7
I20260926 03:35:08.282248 15949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:35:08.295244 15949 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:35:08.296692 15949 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:35:08.314303 15949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4: Generated new TSK 0
I20260926 03:35:08.315302 15949 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:35:08.388286 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43539
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:35:08.783114 15965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:35:08.783483 15965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:35:08.783600 15965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:35:08.792043 15965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:35:08.792199 15965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:35:08.792343 15965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:35:08.803128 15965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43539
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:35:08.804738 15965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:35:08.806290 15965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:35:08.821149 15973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:35:08.821053 15971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:35:08.821188 15970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:35:08.822075 15965 server_base.cc:1034] running on GCE node
I20260926 03:35:08.822778 15965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:35:08.823825 15965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:35:08.825042 15965 hybrid_clock.cc:648] HybridClock initialized: now 1790393708824974 us; error 59 us; skew 500 ppm
I20260926 03:35:08.825392 15965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:35:08 dist-test-slave-dt7p krb5kdc[15859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393708, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:35:08.835103 15965 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:35:08.836901 15965 webserver.cc:460] Webserver started at http://127.0.97.1:34903/ using document root <none> and password file <none>
I20260926 03:35:08.837764 15965 fs_manager.cc:359] Metadata directory not provided
I20260926 03:35:08.837901 15965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:35:08.838265 15965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:35:08.842279 15965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/data/instance:
uuid: "274c847118904ebe8f3142653783b177"
format_stamp: "Formatted at 2026-09-26 03:35:08 on dist-test-slave-dt7p"
server_key: "a79714ed783fa6b911a7325785da2249"
server_key_iv: "7bb97d1072f73851ed64094c80cb3866"
server_key_version: "encryptionkey@0"
I20260926 03:35:08.843186 15965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/wal/instance:
uuid: "274c847118904ebe8f3142653783b177"
format_stamp: "Formatted at 2026-09-26 03:35:08 on dist-test-slave-dt7p"
server_key: "a79714ed783fa6b911a7325785da2249"
server_key_iv: "7bb97d1072f73851ed64094c80cb3866"
server_key_version: "encryptionkey@0"
I20260926 03:35:08.848886 15965 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 03:35:08.852535 15980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:35:08.854049 15965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:35:08.854226 15965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/wal
uuid: "274c847118904ebe8f3142653783b177"
format_stamp: "Formatted at 2026-09-26 03:35:08 on dist-test-slave-dt7p"
server_key: "a79714ed783fa6b911a7325785da2249"
server_key_iv: "7bb97d1072f73851ed64094c80cb3866"
server_key_version: "encryptionkey@0"
I20260926 03:35:08.854434 15965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:35:08.885673 15965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:35:08.890267 15965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:35:08.890599 15965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:35:08.891754 15965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:35:08.893347 15965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:35:08.893425 15965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:35:08.893586 15965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:35:08.893666 15965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:35:08.931342 15965 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45301
I20260926 03:35:08.931389 16092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45301 every 8 connection(s)
I20260926 03:35:08.933111 15965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:35:08.937394 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 15965
I20260926 03:35:08.937681 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:35:08.938647 388 external_mini_cluster.cc:1428] Setting key 8dbd3ec752158c933b8d187daff00863
Sep 26 03:35:08 dist-test-slave-dt7p krb5kdc[15859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393708, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:35:08.955483 16095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:08.936252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:32787 (local address 127.0.97.62:43539)
0926 03:35:08.936827 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0926 03:35:08.936832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:35:08.937986 (+ 1154us) server_negotiation.cc:407] Connection header received
0926 03:35:08.940252 (+ 2266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:35:08.940258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:35:08.940363 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:35:08.940531 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:35:08.942255 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:08.943043 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:08.944807 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:08.945251 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:08.945398 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:08.949459 (+ 4061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:35:08.949464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:35:08.949467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:35:08.949504 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:35:08.952174 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:35:08.952996 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:35:08.953001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:35:08.953004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:35:08.953129 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:35:08.953574 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:35:08.953579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:35:08.953582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:35:08.953887 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:35:08.954066 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 03:35:08.954825 (+ 759us) server_negotiation.cc:299] Negotiation successful
0926 03:35:08.955072 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":293,"threads_started":1}
I20260926 03:35:08.956717 16096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:08.936904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.1:32787)
0926 03:35:08.937794 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:08.937845 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 03:35:08.939955 (+ 2110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:35:08.940818 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:35:08.940828 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:35:08.941405 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:35:08.942067 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:35:08.942085 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:35:08.943209 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:35:08.943213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:35:08.944661 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:35:08.944675 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:35:08.945492 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:35:08.945500 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:35:08.945783 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:08.946542 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:35:08.946567 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:35:08.949207 (+ 2640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:35:08.952331 (+ 3124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:35:08.952340 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:35:08.952357 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:35:08.952868 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:35:08.953234 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:35:08.953248 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:35:08.953252 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:35:08.953455 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:35:08.954014 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:35:08.954025 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:35:08.954414 (+ 389us) client_negotiation.cc:769] Sending connection context
0926 03:35:08.954691 (+ 277us) client_negotiation.cc:240] Negotiation successful
0926 03:35:08.955299 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":392,"threads_started":1}
I20260926 03:35:08.958701 16093 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43539
I20260926 03:35:08.959121 16093 heartbeater.cc:464] Registering TS with master...
I20260926 03:35:08.960027 16093 heartbeater.cc:511] Master 127.0.97.62:43539 requested a full tablet report, sending...
I20260926 03:35:08.962836 15904 ts_manager.cc:194] Registered new tserver with Master: 274c847118904ebe8f3142653783b177 (127.0.97.1:45301)
I20260926 03:35:08.965341 15904 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:32787
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:35:09.001392 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43539
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:35:09.385272 16099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:35:09.385588 16099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:35:09.385695 16099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:35:09.393344 16099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:35:09.393466 16099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:35:09.393591 16099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:35:09.403825 16099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43539
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:35:09.405516 16099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:35:09.407281 16099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:35:09.423377 16104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:35:09.423343 16107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:35:09.423415 16105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:35:09.424340 16099 server_base.cc:1034] running on GCE node
I20260926 03:35:09.425097 16099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:35:09.426195 16099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:35:09.427441 16099 hybrid_clock.cc:648] HybridClock initialized: now 1790393709427375 us; error 60 us; skew 500 ppm
I20260926 03:35:09.427827 16099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:35:09 dist-test-slave-dt7p krb5kdc[15859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:35:09.431557 16099 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:35:09.433331 16099 webserver.cc:460] Webserver started at http://127.0.97.2:40911/ using document root <none> and password file <none>
I20260926 03:35:09.434106 16099 fs_manager.cc:359] Metadata directory not provided
I20260926 03:35:09.434217 16099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:35:09.434525 16099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:35:09.438160 16099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/data/instance:
uuid: "36181f43ae0844ea85f82c508cfdec95"
format_stamp: "Formatted at 2026-09-26 03:35:09 on dist-test-slave-dt7p"
server_key: "6dbeb6dedc2d5978314e0b05541991fc"
server_key_iv: "5e13ce2905cf19692b7d5f56a62c17df"
server_key_version: "encryptionkey@0"
I20260926 03:35:09.439172 16099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/wal/instance:
uuid: "36181f43ae0844ea85f82c508cfdec95"
format_stamp: "Formatted at 2026-09-26 03:35:09 on dist-test-slave-dt7p"
server_key: "6dbeb6dedc2d5978314e0b05541991fc"
server_key_iv: "5e13ce2905cf19692b7d5f56a62c17df"
server_key_version: "encryptionkey@0"
I20260926 03:35:09.445361 16099 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260926 03:35:09.449271 16114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:35:09.450898 16099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:35:09.451066 16099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/wal
uuid: "36181f43ae0844ea85f82c508cfdec95"
format_stamp: "Formatted at 2026-09-26 03:35:09 on dist-test-slave-dt7p"
server_key: "6dbeb6dedc2d5978314e0b05541991fc"
server_key_iv: "5e13ce2905cf19692b7d5f56a62c17df"
server_key_version: "encryptionkey@0"
I20260926 03:35:09.451284 16099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:35:09.468838 16099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:35:09.473572 16099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:35:09.473889 16099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:35:09.474924 16099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:35:09.476377 16099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:35:09.476451 16099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:35:09.476665 16099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:35:09.476718 16099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:35:09.513497 16099 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44503
I20260926 03:35:09.513532 16226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44503 every 8 connection(s)
I20260926 03:35:09.515072 16099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:35:09.520047 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 16099
I20260926 03:35:09.520287 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:35:09.520869 388 external_mini_cluster.cc:1428] Setting key 47949cf4f60773521b64212f7e33bbd6
Sep 26 03:35:09 dist-test-slave-dt7p krb5kdc[15859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:35:09.542214 16229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:09.518092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55717 (local address 127.0.97.62:43539)
0926 03:35:09.518646 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0926 03:35:09.518652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:35:09.519745 (+ 1093us) server_negotiation.cc:407] Connection header received
0926 03:35:09.522189 (+ 2444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:35:09.522194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:35:09.522279 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:35:09.522390 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:35:09.524384 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:09.525825 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:09.528031 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:09.528869 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:09.529022 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:09.534262 (+ 5240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:35:09.534270 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:35:09.534277 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:35:09.534345 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:35:09.538459 (+ 4114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:35:09.539294 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:35:09.539303 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:35:09.539309 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:35:09.539478 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:35:09.539918 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:35:09.539926 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:35:09.539932 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:35:09.540353 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:35:09.540520 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 03:35:09.541356 (+ 836us) server_negotiation.cc:299] Negotiation successful
0926 03:35:09.541647 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":281,"threads_started":1}
I20260926 03:35:09.543103 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:09.518731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.2:55717)
0926 03:35:09.519556 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:09.519599 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 03:35:09.521870 (+ 2271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:35:09.522685 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:35:09.522699 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:35:09.523533 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:35:09.524191 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:35:09.524209 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:35:09.526055 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:35:09.526060 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:35:09.527877 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:35:09.527890 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:35:09.529083 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:35:09.529088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:35:09.529333 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:09.530233 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:35:09.530271 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:35:09.533826 (+ 3555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:35:09.538632 (+ 4806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:35:09.538640 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:35:09.538658 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:35:09.539136 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:35:09.539601 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:35:09.539605 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:35:09.539607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:35:09.539806 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:35:09.540477 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:35:09.540483 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:35:09.540950 (+ 467us) client_negotiation.cc:769] Sending connection context
0926 03:35:09.541236 (+ 286us) client_negotiation.cc:240] Negotiation successful
0926 03:35:09.541696 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":303,"threads_started":1}
I20260926 03:35:09.545115 16227 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43539
I20260926 03:35:09.545506 16227 heartbeater.cc:464] Registering TS with master...
I20260926 03:35:09.546415 16227 heartbeater.cc:511] Master 127.0.97.62:43539 requested a full tablet report, sending...
I20260926 03:35:09.548302 15904 ts_manager.cc:194] Registered new tserver with Master: 36181f43ae0844ea85f82c508cfdec95 (127.0.97.2:44503)
I20260926 03:35:09.549306 15904 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55717
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:35:09.586521 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43539
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:35:09.969744 16093 heartbeater.cc:503] Master 127.0.97.62:43539 was elected leader, sending a full tablet report...
W20260926 03:35:09.976857 16233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:35:09.977258 16233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:35:09.977432 16233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:35:09.985481 16233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:35:09.985661 16233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:35:09.985846 16233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:35:09.996531 16233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43539
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:35:09.998224 16233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:35:09.999819 16233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:35:10.014748 16241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:35:10.014842 16238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:35:10.014883 16239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:35:10.015756 16233 server_base.cc:1034] running on GCE node
I20260926 03:35:10.016470 16233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:35:10.017656 16233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:35:10.018899 16233 hybrid_clock.cc:648] HybridClock initialized: now 1790393710018858 us; error 33 us; skew 500 ppm
I20260926 03:35:10.019253 16233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:35:10 dist-test-slave-dt7p krb5kdc[15859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393710, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:35:10.022783 16233 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:35:10.024466 16233 webserver.cc:460] Webserver started at http://127.0.97.3:44143/ using document root <none> and password file <none>
I20260926 03:35:10.025316 16233 fs_manager.cc:359] Metadata directory not provided
I20260926 03:35:10.025468 16233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:35:10.025821 16233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:35:10.029244 16233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/data/instance:
uuid: "157e59415ba045a5815168d80ffa59b1"
format_stamp: "Formatted at 2026-09-26 03:35:10 on dist-test-slave-dt7p"
server_key: "52c4d483c830ebdffc58ec3f2df12a18"
server_key_iv: "064ec938c8c2670e097170ab00301d57"
server_key_version: "encryptionkey@0"
I20260926 03:35:10.030081 16233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/wal/instance:
uuid: "157e59415ba045a5815168d80ffa59b1"
format_stamp: "Formatted at 2026-09-26 03:35:10 on dist-test-slave-dt7p"
server_key: "52c4d483c830ebdffc58ec3f2df12a18"
server_key_iv: "064ec938c8c2670e097170ab00301d57"
server_key_version: "encryptionkey@0"
I20260926 03:35:10.036026 16233 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260926 03:35:10.040124 16248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:35:10.041774 16233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:35:10.041982 16233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/wal
uuid: "157e59415ba045a5815168d80ffa59b1"
format_stamp: "Formatted at 2026-09-26 03:35:10 on dist-test-slave-dt7p"
server_key: "52c4d483c830ebdffc58ec3f2df12a18"
server_key_iv: "064ec938c8c2670e097170ab00301d57"
server_key_version: "encryptionkey@0"
I20260926 03:35:10.042215 16233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:35:10.054600 16233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:35:10.059404 16233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:35:10.059710 16233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:35:10.060850 16233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:35:10.062319 16233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:35:10.062424 16233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:35:10.062582 16233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:35:10.062659 16233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:35:10.099968 16233 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40249
I20260926 03:35:10.100030 16360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40249 every 8 connection(s)
I20260926 03:35:10.101577 16233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:35:10.106367 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 16233
I20260926 03:35:10.106590 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:35:10.107303 388 external_mini_cluster.cc:1428] Setting key 78eefea9e21ac1f5d672c61507db0032
Sep 26 03:35:10 dist-test-slave-dt7p krb5kdc[15859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393710, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:35:10.123553 16363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:10.104737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42741 (local address 127.0.97.62:43539)
0926 03:35:10.105313 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0926 03:35:10.105318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:35:10.106198 (+ 880us) server_negotiation.cc:407] Connection header received
0926 03:35:10.108528 (+ 2330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:35:10.108535 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:35:10.108671 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:35:10.108806 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:35:10.110521 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:10.111458 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:10.113233 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:10.113700 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:10.113807 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:10.117423 (+ 3616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:35:10.117429 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:35:10.117432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:35:10.117469 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:35:10.120227 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:35:10.120980 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:35:10.120985 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:35:10.120988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:35:10.121141 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:35:10.121680 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:35:10.121684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:35:10.121686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:35:10.122023 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:35:10.122183 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 03:35:10.122980 (+ 797us) server_negotiation.cc:299] Negotiation successful
0926 03:35:10.123189 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":297,"threads_started":1}
I20260926 03:35:10.124809 16364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:10.105226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.3:42741)
0926 03:35:10.106006 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:10.106053 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 03:35:10.108231 (+ 2178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:35:10.109074 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:35:10.109084 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:35:10.109682 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:35:10.110354 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:35:10.110371 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:35:10.111595 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:35:10.111599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:35:10.113097 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:35:10.113110 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:35:10.113896 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:35:10.113901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:35:10.114107 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:10.114830 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:35:10.114854 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:35:10.117205 (+ 2351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:35:10.120345 (+ 3140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:35:10.120352 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:35:10.120365 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:35:10.120855 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:35:10.121293 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:35:10.121298 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:35:10.121300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:35:10.121546 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:35:10.122160 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:35:10.122172 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:35:10.122587 (+ 415us) client_negotiation.cc:769] Sending connection context
0926 03:35:10.122837 (+ 250us) client_negotiation.cc:240] Negotiation successful
0926 03:35:10.123391 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":302,"threads_started":1}
I20260926 03:35:10.126246 16361 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43539
I20260926 03:35:10.126605 16361 heartbeater.cc:464] Registering TS with master...
I20260926 03:35:10.127415 16361 heartbeater.cc:511] Master 127.0.97.62:43539 requested a full tablet report, sending...
I20260926 03:35:10.128993 15904 ts_manager.cc:194] Registered new tserver with Master: 157e59415ba045a5815168d80ffa59b1 (127.0.97.3:40249)
I20260926 03:35:10.129889 15904 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42741
I20260926 03:35:10.135634 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:35:10.155066 16363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:10.140977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60362 (local address 127.0.97.62:43539)
0926 03:35:10.141163 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 03:35:10.141169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:35:10.141645 (+ 476us) server_negotiation.cc:407] Connection header received
0926 03:35:10.141885 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:35:10.141889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:35:10.141971 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:35:10.142067 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:35:10.143629 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:10.144459 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:10.145962 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:10.146448 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:10.146550 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:10.149518 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:35:10.149524 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:35:10.149527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:35:10.149565 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:35:10.152420 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:35:10.153116 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:35:10.153120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:35:10.153123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:35:10.153244 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:35:10.153574 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:35:10.153577 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:35:10.153579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:35:10.153942 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:35:10.154150 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 03:35:10.154412 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 03:35:10.154617 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 03:35:10.159837 15904 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60362:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:35:10.167443 15904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:35:10.225833 16374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:10.181838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44503 (local address 127.0.0.1:52222)
0926 03:35:10.211767 (+ 29929us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:10.211798 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 03:35:10.211983 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:35:10.216775 (+ 4792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:35:10.216780 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:35:10.216810 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:35:10.217477 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:35:10.217491 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:35:10.219836 (+ 2345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:35:10.219842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:35:10.222606 (+ 2764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:35:10.222621 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:35:10.224652 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:35:10.224658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:35:10.224885 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:10.224938 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 03:35:10.225068 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 03:35:10.225300 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28034,"mutex_wait_us":27,"thread_start_us":20330,"threads_started":1}
I20260926 03:35:10.226505 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:10.182940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57070 (local address 127.0.97.3:40249)
0926 03:35:10.214489 (+ 31549us) server_negotiation.cc:206] Beginning negotiation
0926 03:35:10.214506 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 03:35:10.215758 (+ 1252us) server_negotiation.cc:407] Connection header received
0926 03:35:10.217126 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:35:10.217134 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:35:10.217394 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:35:10.217610 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:35:10.218658 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:10.220186 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:10.223036 (+ 2850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:10.224556 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:10.224724 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:10.225021 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0926 03:35:10.225932 (+ 911us) server_negotiation.cc:299] Negotiation successful
0926 03:35:10.226113 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31232,"thread_start_us":31595,"threads_started":1}
I20260926 03:35:10.227125 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:10.182300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40249 (local address 127.0.0.1:57070)
0926 03:35:10.215526 (+ 33226us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:10.215552 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 03:35:10.216867 (+ 1315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:35:10.217950 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:35:10.217957 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:35:10.217990 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:35:10.218445 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:35:10.218460 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:35:10.220389 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:35:10.220393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:35:10.222853 (+ 2460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:35:10.222867 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:35:10.225184 (+ 2317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:35:10.225191 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:35:10.225385 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:10.225418 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 03:35:10.225514 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 03:35:10.225671 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33095,"mutex_wait_us":1,"thread_start_us":33293,"threads_started":1}
I20260926 03:35:10.227178 16375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:10.181662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45301 (local address 127.0.0.1:53858)
0926 03:35:10.204003 (+ 22341us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:10.204093 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0926 03:35:10.204379 (+ 286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:35:10.216616 (+ 12237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:35:10.216625 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:35:10.216825 (+ 200us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:35:10.218048 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:35:10.218071 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:35:10.221619 (+ 3548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:35:10.221624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:35:10.223810 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:35:10.223825 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:35:10.225777 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:35:10.225785 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:35:10.225958 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:10.226249 (+ 291us) client_negotiation.cc:769] Sending connection context
0926 03:35:10.226316 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 03:35:10.226451 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20669,"mutex_wait_us":57,"spinlock_wait_cycles":335616,"thread_start_us":27952,"threads_started":1}
I20260926 03:35:10.227396 16376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:10.182198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52222 (local address 127.0.97.2:44503)
0926 03:35:10.216142 (+ 33944us) server_negotiation.cc:206] Beginning negotiation
0926 03:35:10.216151 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:35:10.216180 (+ 29us) server_negotiation.cc:407] Connection header received
0926 03:35:10.216304 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:35:10.216311 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:35:10.216520 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:35:10.217702 (+ 1182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:35:10.217950 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:10.219679 (+ 1729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:10.222830 (+ 3151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:10.224407 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:10.226569 (+ 2162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:10.226765 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 03:35:10.226838 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 03:35:10.227026 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33764,"thread_start_us":33605,"threads_started":1}
I20260926 03:35:10.228688 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:10.182380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53858 (local address 127.0.97.1:45301)
0926 03:35:10.215738 (+ 33358us) server_negotiation.cc:206] Beginning negotiation
0926 03:35:10.215751 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:35:10.215793 (+ 42us) server_negotiation.cc:407] Connection header received
0926 03:35:10.215937 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:35:10.215948 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:35:10.216302 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:35:10.216565 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:35:10.218320 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:10.221451 (+ 3131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:10.224023 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:10.225537 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:10.227871 (+ 2334us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:10.228091 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 03:35:10.228155 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 03:35:10.228326 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33101,"thread_start_us":32907,"threads_started":1}
I20260926 03:35:10.232295 16028 tablet_service.cc:1467] Processing CreateTablet for tablet 1d6b5221cba648c4b8fcd744a05d9521 (DEFAULT_TABLE table=test-table [id=bcb4db70808640079be818cc0ec682ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:35:10.232100 16162 tablet_service.cc:1467] Processing CreateTablet for tablet 1d6b5221cba648c4b8fcd744a05d9521 (DEFAULT_TABLE table=test-table [id=bcb4db70808640079be818cc0ec682ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:35:10.232092 16296 tablet_service.cc:1467] Processing CreateTablet for tablet 1d6b5221cba648c4b8fcd744a05d9521 (DEFAULT_TABLE table=test-table [id=bcb4db70808640079be818cc0ec682ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:35:10.233857 16028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d6b5221cba648c4b8fcd744a05d9521. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:35:10.233867 16296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d6b5221cba648c4b8fcd744a05d9521. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:35:10.233891 16162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d6b5221cba648c4b8fcd744a05d9521. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:35:10.247642 16380 tablet_bootstrap.cc:492] T 1d6b5221cba648c4b8fcd744a05d9521 P 157e59415ba045a5815168d80ffa59b1: Bootstrap starting.
I20260926 03:35:10.250168 16382 tablet_bootstrap.cc:492] T 1d6b5221cba648c4b8fcd744a05d9521 P 274c847118904ebe8f3142653783b177: Bootstrap starting.
I20260926 03:35:10.252125 16381 tablet_bootstrap.cc:492] T 1d6b5221cba648c4b8fcd744a05d9521 P 36181f43ae0844ea85f82c508cfdec95: Bootstrap starting.
I20260926 03:35:10.252161 16380 tablet_bootstrap.cc:654] T 1d6b5221cba648c4b8fcd744a05d9521 P 157e59415ba045a5815168d80ffa59b1: Neither blocks nor log segments found. Creating new log.
I20260926 03:35:10.253744 16380 log.cc:824] T 1d6b5221cba648c4b8fcd744a05d9521 P 157e59415ba045a5815168d80ffa59b1: Log is configured to *not* fsync() on all Append() calls
I20260926 03:35:10.254333 16382 tablet_bootstrap.cc:654] T 1d6b5221cba648c4b8fcd744a05d9521 P 274c847118904ebe8f3142653783b177: Neither blocks nor log segments found. Creating new log.
I20260926 03:35:10.255986 16382 log.cc:824] T 1d6b5221cba648c4b8fcd744a05d9521 P 274c847118904ebe8f3142653783b177: Log is configured to *not* fsync() on all Append() calls
I20260926 03:35:10.257026 16381 tablet_bootstrap.cc:654] T 1d6b5221cba648c4b8fcd744a05d9521 P 36181f43ae0844ea85f82c508cfdec95: Neither blocks nor log segments found. Creating new log.
I20260926 03:35:10.258005 16380 tablet_bootstrap.cc:492] T 1d6b5221cba648c4b8fcd744a05d9521 P 157e59415ba045a5815168d80ffa59b1: No bootstrap required, opened a new log
I20260926 03:35:10.258338 16380 ts_tablet_manager.cc:1397] T 1d6b5221cba648c4b8fcd744a05d9521 P 157e59415ba045a5815168d80ffa59b1: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260926 03:35:10.258524 16381 log.cc:824] T 1d6b5221cba648c4b8fcd744a05d9521 P 36181f43ae0844ea85f82c508cfdec95: Log is configured to *not* fsync() on all Append() calls
I20260926 03:35:10.259286 16382 tablet_bootstrap.cc:492] T 1d6b5221cba648c4b8fcd744a05d9521 P 274c847118904ebe8f3142653783b177: No bootstrap required, opened a new log
I20260926 03:35:10.259624 16382 ts_tablet_manager.cc:1397] T 1d6b5221cba648c4b8fcd744a05d9521 P 274c847118904ebe8f3142653783b177: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260926 03:35:10.261569 16381 tablet_bootstrap.cc:492] T 1d6b5221cba648c4b8fcd744a05d9521 P 36181f43ae0844ea85f82c508cfdec95: No bootstrap required, opened a new log
I20260926 03:35:10.261849 16381 ts_tablet_manager.cc:1397] T 1d6b5221cba648c4b8fcd744a05d9521 P 36181f43ae0844ea85f82c508cfdec95: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260926 03:35:10.269917 16381 raft_consensus.cc:348] T 1d6b5221cba648c4b8fcd744a05d9521 P 36181f43ae0844ea85f82c508cfdec95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274c847118904ebe8f3142653783b177" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45301 } } peers { permanent_uuid: "36181f43ae0844ea85f82c508cfdec95" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44503 } } peers { permanent_uuid: "157e59415ba045a5815168d80ffa59b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40249 } }
I20260926 03:35:10.270392 16381 raft_consensus.cc:374] T 1d6b5221cba648c4b8fcd744a05d9521 P 36181f43ae0844ea85f82c508cfdec95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:35:10.270526 16381 raft_consensus.cc:729] T 1d6b5221cba648c4b8fcd744a05d9521 P 36181f43ae0844ea85f82c508cfdec95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36181f43ae0844ea85f82c508cfdec95, State: Initialized, Role: FOLLOWER
I20260926 03:35:10.271250 16381 consensus_queue.cc:260] T 1d6b5221cba648c4b8fcd744a05d9521 P 36181f43ae0844ea85f82c508cfdec95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274c847118904ebe8f3142653783b177" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45301 } } peers { permanent_uuid: "36181f43ae0844ea85f82c508cfdec95" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44503 } } peers { permanent_uuid: "157e59415ba045a5815168d80ffa59b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40249 } }
I20260926 03:35:10.271193 16380 raft_consensus.cc:348] T 1d6b5221cba648c4b8fcd744a05d9521 P 157e59415ba045a5815168d80ffa59b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274c847118904ebe8f3142653783b177" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45301 } } peers { permanent_uuid: "36181f43ae0844ea85f82c508cfdec95" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44503 } } peers { permanent_uuid: "157e59415ba045a5815168d80ffa59b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40249 } }
I20260926 03:35:10.271201 16382 raft_consensus.cc:348] T 1d6b5221cba648c4b8fcd744a05d9521 P 274c847118904ebe8f3142653783b177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274c847118904ebe8f3142653783b177" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45301 } } peers { permanent_uuid: "36181f43ae0844ea85f82c508cfdec95" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44503 } } peers { permanent_uuid: "157e59415ba045a5815168d80ffa59b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40249 } }
I20260926 03:35:10.271656 16382 raft_consensus.cc:374] T 1d6b5221cba648c4b8fcd744a05d9521 P 274c847118904ebe8f3142653783b177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:35:10.271698 16380 raft_consensus.cc:374] T 1d6b5221cba648c4b8fcd744a05d9521 P 157e59415ba045a5815168d80ffa59b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:35:10.271778 16382 raft_consensus.cc:729] T 1d6b5221cba648c4b8fcd744a05d9521 P 274c847118904ebe8f3142653783b177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 274c847118904ebe8f3142653783b177, State: Initialized, Role: FOLLOWER
I20260926 03:35:10.271793 16380 raft_consensus.cc:729] T 1d6b5221cba648c4b8fcd744a05d9521 P 157e59415ba045a5815168d80ffa59b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 157e59415ba045a5815168d80ffa59b1, State: Initialized, Role: FOLLOWER
I20260926 03:35:10.272415 16382 consensus_queue.cc:260] T 1d6b5221cba648c4b8fcd744a05d9521 P 274c847118904ebe8f3142653783b177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274c847118904ebe8f3142653783b177" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45301 } } peers { permanent_uuid: "36181f43ae0844ea85f82c508cfdec95" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44503 } } peers { permanent_uuid: "157e59415ba045a5815168d80ffa59b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40249 } }
I20260926 03:35:10.272416 16380 consensus_queue.cc:260] T 1d6b5221cba648c4b8fcd744a05d9521 P 157e59415ba045a5815168d80ffa59b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274c847118904ebe8f3142653783b177" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45301 } } peers { permanent_uuid: "36181f43ae0844ea85f82c508cfdec95" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44503 } } peers { permanent_uuid: "157e59415ba045a5815168d80ffa59b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40249 } }
I20260926 03:35:10.273524 16227 heartbeater.cc:503] Master 127.0.97.62:43539 was elected leader, sending a full tablet report...
I20260926 03:35:10.274159 16381 ts_tablet_manager.cc:1428] T 1d6b5221cba648c4b8fcd744a05d9521 P 36181f43ae0844ea85f82c508cfdec95: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260926 03:35:10.277433 16361 heartbeater.cc:503] Master 127.0.97.62:43539 was elected leader, sending a full tablet report...
I20260926 03:35:10.280454 16382 ts_tablet_manager.cc:1428] T 1d6b5221cba648c4b8fcd744a05d9521 P 274c847118904ebe8f3142653783b177: Time spent starting tablet: real 0.019s user 0.014s sys 0.004s
I20260926 03:35:10.282159 16380 ts_tablet_manager.cc:1428] T 1d6b5221cba648c4b8fcd744a05d9521 P 157e59415ba045a5815168d80ffa59b1: Time spent starting tablet: real 0.024s user 0.011s sys 0.006s
I20260926 03:35:10.295064 16387 raft_consensus.cc:482] T 1d6b5221cba648c4b8fcd744a05d9521 P 157e59415ba045a5815168d80ffa59b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:35:10.295435 16387 raft_consensus.cc:504] T 1d6b5221cba648c4b8fcd744a05d9521 P 157e59415ba045a5815168d80ffa59b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274c847118904ebe8f3142653783b177" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45301 } } peers { permanent_uuid: "36181f43ae0844ea85f82c508cfdec95" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44503 } } peers { permanent_uuid: "157e59415ba045a5815168d80ffa59b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40249 } }
I20260926 03:35:10.297619 16387 leader_election.cc:288] T 1d6b5221cba648c4b8fcd744a05d9521 P 157e59415ba045a5815168d80ffa59b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 274c847118904ebe8f3142653783b177 (127.0.97.1:45301), 36181f43ae0844ea85f82c508cfdec95 (127.0.97.2:44503)
I20260926 03:35:10.305723 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:10.298134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37009 (local address 127.0.97.1:45301)
0926 03:35:10.298447 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0926 03:35:10.298457 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:35:10.298554 (+ 97us) server_negotiation.cc:407] Connection header received
0926 03:35:10.298827 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:35:10.298833 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:35:10.298929 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:35:10.299041 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:35:10.299552 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:10.300756 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:10.303288 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:10.304655 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:10.304812 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:10.304955 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 03:35:10.305279 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 03:35:10.305432 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260926 03:35:10.305739 16364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:10.298061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45301 (local address 127.0.97.3:37009)
0926 03:35:10.298437 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:10.298461 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:35:10.298622 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:35:10.299081 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:35:10.299087 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:35:10.299133 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:35:10.299420 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:35:10.299429 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:35:10.300887 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:35:10.300891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:35:10.303112 (+ 2221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:35:10.303128 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:35:10.304860 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:35:10.304866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:35:10.305081 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:10.305123 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 03:35:10.305232 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 03:35:10.305393 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":74,"spinlock_wait_cycles":25984}
I20260926 03:35:10.305992 16376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:10.298161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36411 (local address 127.0.97.2:44503)
0926 03:35:10.298424 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 03:35:10.298432 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:35:10.299201 (+ 769us) server_negotiation.cc:407] Connection header received
0926 03:35:10.299396 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:35:10.299399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:35:10.299468 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:35:10.299588 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:35:10.300078 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:10.301111 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:10.303510 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:10.304858 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:10.305003 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:10.305121 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 03:35:10.305553 (+ 432us) server_negotiation.cc:299] Negotiation successful
0926 03:35:10.305696 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 03:35:10.306640 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:10.298222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44503 (local address 127.0.97.3:36411)
0926 03:35:10.299080 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:10.299095 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:35:10.299239 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:35:10.299601 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:35:10.299604 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:35:10.299624 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:35:10.299905 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:35:10.299912 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:35:10.301267 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:35:10.301272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:35:10.303348 (+ 2076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:35:10.303358 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:35:10.305142 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:35:10.305151 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:35:10.305355 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:10.305390 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 03:35:10.306081 (+ 691us) client_negotiation.cc:240] Negotiation successful
0926 03:35:10.306281 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":532,"threads_started":1}
I20260926 03:35:10.307381 16048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d6b5221cba648c4b8fcd744a05d9521" candidate_uuid: "157e59415ba045a5815168d80ffa59b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274c847118904ebe8f3142653783b177" is_pre_election: true
I20260926 03:35:10.307889 16182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d6b5221cba648c4b8fcd744a05d9521" candidate_uuid: "157e59415ba045a5815168d80ffa59b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36181f43ae0844ea85f82c508cfdec95" is_pre_election: true
I20260926 03:35:10.308214 16048 raft_consensus.cc:2454] T 1d6b5221cba648c4b8fcd744a05d9521 P 274c847118904ebe8f3142653783b177 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 157e59415ba045a5815168d80ffa59b1 in term 0.
I20260926 03:35:10.308557 16182 raft_consensus.cc:2454] T 1d6b5221cba648c4b8fcd744a05d9521 P 36181f43ae0844ea85f82c508cfdec95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 157e59415ba045a5815168d80ffa59b1 in term 0.
I20260926 03:35:10.309233 16252 leader_election.cc:302] T 1d6b5221cba648c4b8fcd744a05d9521 P 157e59415ba045a5815168d80ffa59b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 157e59415ba045a5815168d80ffa59b1, 274c847118904ebe8f3142653783b177; no voters:
I20260926 03:35:10.309679 16387 raft_consensus.cc:2781] T 1d6b5221cba648c4b8fcd744a05d9521 P 157e59415ba045a5815168d80ffa59b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:35:10.309862 16387 raft_consensus.cc:482] T 1d6b5221cba648c4b8fcd744a05d9521 P 157e59415ba045a5815168d80ffa59b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:35:10.310000 16387 raft_consensus.cc:3037] T 1d6b5221cba648c4b8fcd744a05d9521 P 157e59415ba045a5815168d80ffa59b1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:35:10.312819 16387 raft_consensus.cc:504] T 1d6b5221cba648c4b8fcd744a05d9521 P 157e59415ba045a5815168d80ffa59b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274c847118904ebe8f3142653783b177" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45301 } } peers { permanent_uuid: "36181f43ae0844ea85f82c508cfdec95" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44503 } } peers { permanent_uuid: "157e59415ba045a5815168d80ffa59b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40249 } }
I20260926 03:35:10.313730 16387 leader_election.cc:288] T 1d6b5221cba648c4b8fcd744a05d9521 P 157e59415ba045a5815168d80ffa59b1 [CANDIDATE]: Term 1 election: Requested vote from peers 274c847118904ebe8f3142653783b177 (127.0.97.1:45301), 36181f43ae0844ea85f82c508cfdec95 (127.0.97.2:44503)
I20260926 03:35:10.314220 16048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d6b5221cba648c4b8fcd744a05d9521" candidate_uuid: "157e59415ba045a5815168d80ffa59b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274c847118904ebe8f3142653783b177"
I20260926 03:35:10.314323 16182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d6b5221cba648c4b8fcd744a05d9521" candidate_uuid: "157e59415ba045a5815168d80ffa59b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36181f43ae0844ea85f82c508cfdec95"
I20260926 03:35:10.314468 16048 raft_consensus.cc:3037] T 1d6b5221cba648c4b8fcd744a05d9521 P 274c847118904ebe8f3142653783b177 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:35:10.314595 16182 raft_consensus.cc:3037] T 1d6b5221cba648c4b8fcd744a05d9521 P 36181f43ae0844ea85f82c508cfdec95 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:35:10.317185 16182 raft_consensus.cc:2454] T 1d6b5221cba648c4b8fcd744a05d9521 P 36181f43ae0844ea85f82c508cfdec95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 157e59415ba045a5815168d80ffa59b1 in term 1.
I20260926 03:35:10.317210 16048 raft_consensus.cc:2454] T 1d6b5221cba648c4b8fcd744a05d9521 P 274c847118904ebe8f3142653783b177 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 157e59415ba045a5815168d80ffa59b1 in term 1.
I20260926 03:35:10.317656 16250 leader_election.cc:302] T 1d6b5221cba648c4b8fcd744a05d9521 P 157e59415ba045a5815168d80ffa59b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 157e59415ba045a5815168d80ffa59b1, 36181f43ae0844ea85f82c508cfdec95; no voters:
I20260926 03:35:10.318034 16387 raft_consensus.cc:2781] T 1d6b5221cba648c4b8fcd744a05d9521 P 157e59415ba045a5815168d80ffa59b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:35:10.318574 16387 raft_consensus.cc:686] T 1d6b5221cba648c4b8fcd744a05d9521 P 157e59415ba045a5815168d80ffa59b1 [term 1 LEADER]: Becoming Leader. State: Replica: 157e59415ba045a5815168d80ffa59b1, State: Running, Role: LEADER
I20260926 03:35:10.319231 16387 consensus_queue.cc:237] T 1d6b5221cba648c4b8fcd744a05d9521 P 157e59415ba045a5815168d80ffa59b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274c847118904ebe8f3142653783b177" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45301 } } peers { permanent_uuid: "36181f43ae0844ea85f82c508cfdec95" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44503 } } peers { permanent_uuid: "157e59415ba045a5815168d80ffa59b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40249 } }
I20260926 03:35:10.324892 15903 catalog_manager.cc:5495] T 1d6b5221cba648c4b8fcd744a05d9521 P 157e59415ba045a5815168d80ffa59b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 157e59415ba045a5815168d80ffa59b1 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "157e59415ba045a5815168d80ffa59b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274c847118904ebe8f3142653783b177" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36181f43ae0844ea85f82c508cfdec95" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "157e59415ba045a5815168d80ffa59b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40249 } health_report { overall_health: HEALTHY } } }
I20260926 03:35:10.350449 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:10.344634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57078 (local address 127.0.97.3:40249)
0926 03:35:10.344855 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 03:35:10.344861 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:35:10.344880 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:35:10.345101 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:35:10.345106 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:35:10.345205 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:35:10.345347 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:35:10.345773 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:10.346683 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:10.348200 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:10.348693 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:10.348804 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:10.349270 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:35:10.349919 (+ 649us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:35:10.350039 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 03:35:10.350122 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 03:35:10.350200 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 03:35:10.353456 16362 tablet.cc:2369] T 1d6b5221cba648c4b8fcd744a05d9521 P 157e59415ba045a5815168d80ffa59b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:35:10.361171 16048 raft_consensus.cc:1261] T 1d6b5221cba648c4b8fcd744a05d9521 P 274c847118904ebe8f3142653783b177 [term 1 FOLLOWER]: Refusing update from remote peer 157e59415ba045a5815168d80ffa59b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:35:10.361186 16182 raft_consensus.cc:1261] T 1d6b5221cba648c4b8fcd744a05d9521 P 36181f43ae0844ea85f82c508cfdec95 [term 1 FOLLOWER]: Refusing update from remote peer 157e59415ba045a5815168d80ffa59b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:35:10.362289 16387 consensus_queue.cc:1035] T 1d6b5221cba648c4b8fcd744a05d9521 P 157e59415ba045a5815168d80ffa59b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "274c847118904ebe8f3142653783b177" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:35:10.362697 16390 consensus_queue.cc:1035] T 1d6b5221cba648c4b8fcd744a05d9521 P 157e59415ba045a5815168d80ffa59b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36181f43ae0844ea85f82c508cfdec95" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:35:10.376406 16393 mvcc.cc:205] Tried to move back new op lower bound from 7333452637626880000 to 7333452637471985664. Current Snapshot: MvccSnapshot[applied={T|T < 7333452637626880000}]
I20260926 03:35:10.392872 15903 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:35:10.393191 15903 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:35:10.396906 15903 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4: Sending DeleteTablet for 3 replicas of tablet 1d6b5221cba648c4b8fcd744a05d9521
I20260926 03:35:10.398262 16162 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d6b5221cba648c4b8fcd744a05d9521 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:35:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52222
I20260926 03:35:10.398437 16296 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d6b5221cba648c4b8fcd744a05d9521 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:35:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57070
I20260926 03:35:10.398494 16028 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d6b5221cba648c4b8fcd744a05d9521 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:35:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53858
I20260926 03:35:10.399678 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 15965
I20260926 03:35:10.416267 16405 tablet_replica.cc:330] T 1d6b5221cba648c4b8fcd744a05d9521 P 157e59415ba045a5815168d80ffa59b1: stopping tablet replica
I20260926 03:35:10.417049 16406 tablet_replica.cc:330] T 1d6b5221cba648c4b8fcd744a05d9521 P 274c847118904ebe8f3142653783b177: stopping tablet replica
I20260926 03:35:10.417548 16404 tablet_replica.cc:330] T 1d6b5221cba648c4b8fcd744a05d9521 P 36181f43ae0844ea85f82c508cfdec95: stopping tablet replica
I20260926 03:35:10.417678 16406 raft_consensus.cc:2229] T 1d6b5221cba648c4b8fcd744a05d9521 P 274c847118904ebe8f3142653783b177 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:35:10.418354 16405 raft_consensus.cc:2229] T 1d6b5221cba648c4b8fcd744a05d9521 P 157e59415ba045a5815168d80ffa59b1 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:35:10.418540 16404 raft_consensus.cc:2229] T 1d6b5221cba648c4b8fcd744a05d9521 P 36181f43ae0844ea85f82c508cfdec95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:35:10.419255 16404 raft_consensus.cc:2258] T 1d6b5221cba648c4b8fcd744a05d9521 P 36181f43ae0844ea85f82c508cfdec95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:35:10.420027 16405 raft_consensus.cc:2258] T 1d6b5221cba648c4b8fcd744a05d9521 P 157e59415ba045a5815168d80ffa59b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:35:10.421139 16406 raft_consensus.cc:2258] T 1d6b5221cba648c4b8fcd744a05d9521 P 274c847118904ebe8f3142653783b177 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:35:10.422622 16404 ts_tablet_manager.cc:1905] T 1d6b5221cba648c4b8fcd744a05d9521 P 36181f43ae0844ea85f82c508cfdec95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:35:10.422821 16405 ts_tablet_manager.cc:1905] T 1d6b5221cba648c4b8fcd744a05d9521 P 157e59415ba045a5815168d80ffa59b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:35:10.423363 16406 ts_tablet_manager.cc:1905] T 1d6b5221cba648c4b8fcd744a05d9521 P 274c847118904ebe8f3142653783b177: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 03:35:10 dist-test-slave-dt7p krb5kdc[15859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:35:10.430528 16404 ts_tablet_manager.cc:1918] T 1d6b5221cba648c4b8fcd744a05d9521 P 36181f43ae0844ea85f82c508cfdec95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:35:10.430855 16404 log.cc:1196] T 1d6b5221cba648c4b8fcd744a05d9521 P 36181f43ae0844ea85f82c508cfdec95: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/wal/wals/1d6b5221cba648c4b8fcd744a05d9521
I20260926 03:35:10.430934 16405 ts_tablet_manager.cc:1918] T 1d6b5221cba648c4b8fcd744a05d9521 P 157e59415ba045a5815168d80ffa59b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:35:10.430984 16406 ts_tablet_manager.cc:1918] T 1d6b5221cba648c4b8fcd744a05d9521 P 274c847118904ebe8f3142653783b177: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:35:10.431139 16406 log.cc:1196] T 1d6b5221cba648c4b8fcd744a05d9521 P 274c847118904ebe8f3142653783b177: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/wal/wals/1d6b5221cba648c4b8fcd744a05d9521
I20260926 03:35:10.431245 16405 log.cc:1196] T 1d6b5221cba648c4b8fcd744a05d9521 P 157e59415ba045a5815168d80ffa59b1: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/wal/wals/1d6b5221cba648c4b8fcd744a05d9521
I20260926 03:35:10.431545 16404 ts_tablet_manager.cc:1939] T 1d6b5221cba648c4b8fcd744a05d9521 P 36181f43ae0844ea85f82c508cfdec95: Deleting consensus metadata
I20260926 03:35:10.431816 16406 ts_tablet_manager.cc:1939] T 1d6b5221cba648c4b8fcd744a05d9521 P 274c847118904ebe8f3142653783b177: Deleting consensus metadata
I20260926 03:35:10.432849 16405 ts_tablet_manager.cc:1939] T 1d6b5221cba648c4b8fcd744a05d9521 P 157e59415ba045a5815168d80ffa59b1: Deleting consensus metadata
I20260926 03:35:10.434026 15891 catalog_manager.cc:4841] TS 274c847118904ebe8f3142653783b177 (127.0.97.1:45301): tablet 1d6b5221cba648c4b8fcd744a05d9521 (table test-table [id=bcb4db70808640079be818cc0ec682ae]) successfully deleted
I20260926 03:35:10.434830 15889 catalog_manager.cc:4841] TS 36181f43ae0844ea85f82c508cfdec95 (127.0.97.2:44503): tablet 1d6b5221cba648c4b8fcd744a05d9521 (table test-table [id=bcb4db70808640079be818cc0ec682ae]) successfully deleted
I20260926 03:35:10.434890 15888 catalog_manager.cc:4841] TS 157e59415ba045a5815168d80ffa59b1 (127.0.97.3:40249): tablet 1d6b5221cba648c4b8fcd744a05d9521 (table test-table [id=bcb4db70808640079be818cc0ec682ae]) successfully deleted
I20260926 03:35:10.440868 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:10.401350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53870 (local address 127.0.97.1:45301)
0926 03:35:10.401609 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 03:35:10.401619 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:35:10.407271 (+ 5652us) server_negotiation.cc:407] Connection header received
0926 03:35:10.408459 (+ 1188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:35:10.408467 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:35:10.408635 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:35:10.408734 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:35:10.414486 (+ 5752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:10.416042 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:10.419068 (+ 3026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:10.419718 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:10.419821 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:10.433484 (+ 13663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:35:10.433497 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:35:10.433518 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:35:10.433567 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:35:10.437468 (+ 3901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:35:10.438504 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:35:10.438512 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:35:10.438515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:35:10.438665 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:35:10.439125 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:35:10.439128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:35:10.439130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:35:10.439539 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:35:10.439798 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0926 03:35:10.440243 (+ 445us) server_negotiation.cc:299] Negotiation successful
0926 03:35:10.440431 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 03:35:10.441740 16088 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:35:10.540035 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 15965
I20260926 03:35:10.563022 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 16099
Sep 26 03:35:10 dist-test-slave-dt7p krb5kdc[15859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:35:10.582824 16376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:10.564197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52232 (local address 127.0.97.2:44503)
0926 03:35:10.564429 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 03:35:10.564436 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:35:10.564459 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:35:10.564682 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:35:10.564689 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:35:10.564788 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:35:10.564919 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:35:10.566493 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:10.567757 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:10.569308 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:10.569949 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:10.570083 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:10.575255 (+ 5172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:35:10.575292 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:35:10.575323 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:35:10.575388 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:35:10.579666 (+ 4278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:35:10.580371 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:35:10.580383 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:35:10.580387 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:35:10.580544 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:35:10.580940 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:35:10.580945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:35:10.580949 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:35:10.581515 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:35:10.581832 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0926 03:35:10.582147 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 03:35:10.582410 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 03:35:10.583730 16222 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:35:10.681694 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 16099
I20260926 03:35:10.702251 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 16233
Sep 26 03:35:10 dist-test-slave-dt7p krb5kdc[15859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:35:10.722812 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:10.703260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57082 (local address 127.0.97.3:40249)
0926 03:35:10.703513 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 03:35:10.703519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:35:10.703626 (+ 107us) server_negotiation.cc:407] Connection header received
0926 03:35:10.703833 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:35:10.703838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:35:10.703920 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:35:10.704009 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:35:10.706176 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:10.707078 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:10.709456 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:10.709969 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:10.710085 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:10.716012 (+ 5927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:35:10.716025 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:35:10.716049 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:35:10.716093 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:35:10.719636 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:35:10.720405 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:35:10.720413 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:35:10.720416 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:35:10.720628 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:35:10.721069 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:35:10.721080 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:35:10.721083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:35:10.721510 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:35:10.721900 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0926 03:35:10.722129 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 03:35:10.722379 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260926 03:35:10.723663 16356 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:35:10.862435 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 16233
I20260926 03:35:10.885329 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 15872
I20260926 03:35:10.886636 15934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:35:11.015298 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 15872
I20260926 03:35:31.041587 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43539
--webserver_interface=127.0.97.62
--webserver_port=42683
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:35:31.423113 16412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:35:31.423496 16412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:35:31.423668 16412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:35:31.431617 16412 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 03:35:31.431728 16412 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 03:35:31.431802 16412 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 03:35:31.431881 16412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:35:31.431979 16412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:35:31.432117 16412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:35:31.432190 16412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:35:31.432286 16412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:35:31.443313 16412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=42683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:35:31.445034 16412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:35:31.446650 16412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:35:31.472961 16421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:35:31.472914 16418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:35:31.472934 16419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:35:31.473968 16412 server_base.cc:1034] running on GCE node
I20260926 03:35:31.475025 16412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:35:31.477043 16412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:35:31.478314 16412 hybrid_clock.cc:648] HybridClock initialized: now 1790393731478238 us; error 46 us; skew 500 ppm
I20260926 03:35:31.478749 16412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:35:31 dist-test-slave-dt7p krb5kdc[15859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:35:31.490449 16412 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:35:31.492492 16412 webserver.cc:460] Webserver started at http://127.0.97.62:42683/ using document root <none> and password file <none>
I20260926 03:35:31.493459 16412 fs_manager.cc:359] Metadata directory not provided
I20260926 03:35:31.493623 16412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:35:31.500715 16412 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 03:35:31.504006 16428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:35:31.505563 16412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 03:35:31.505774 16412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "5e2037cbf110405d90315bdb3e5d87c4"
format_stamp: "Formatted at 2026-09-26 03:35:08 on dist-test-slave-dt7p"
server_key: "dd02b3f03e7d752593de335f20be849a"
server_key_iv: "c7fb82582765c1f9b337292ae70e5c36"
server_key_version: "encryptionkey@0"
I20260926 03:35:31.506927 16412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:35:31.520712 16412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:35:31.525698 16412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:35:31.526024 16412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:35:31.547789 16412 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43539
I20260926 03:35:31.547811 16479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43539 every 8 connection(s)
I20260926 03:35:31.549779 16412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:35:31.558084 16480 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:35:31.559685 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 16412
I20260926 03:35:31.560732 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:45301
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:43539
--webserver_port=34903
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:35:31.562018 16480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4: Bootstrap starting.
I20260926 03:35:31.585812 16480 log.cc:824] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4: Log is configured to *not* fsync() on all Append() calls
I20260926 03:35:31.599802 16480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:35:31.600436 16480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4: Bootstrap complete.
I20260926 03:35:31.610572 16480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e2037cbf110405d90315bdb3e5d87c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43539 } }
I20260926 03:35:31.611702 16480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e2037cbf110405d90315bdb3e5d87c4, State: Initialized, Role: FOLLOWER
I20260926 03:35:31.612464 16480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e2037cbf110405d90315bdb3e5d87c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43539 } }
I20260926 03:35:31.612812 16480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:35:31.612951 16480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:35:31.613158 16480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [term 1 FOLLOWER]: Advancing to term 2
I20260926 03:35:31.617738 16480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e2037cbf110405d90315bdb3e5d87c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43539 } }
I20260926 03:35:31.618376 16480 leader_election.cc:302] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e2037cbf110405d90315bdb3e5d87c4; no voters:
I20260926 03:35:31.619155 16480 leader_election.cc:288] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 03:35:31.619378 16484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 03:35:31.620909 16484 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [term 2 LEADER]: Becoming Leader. State: Replica: 5e2037cbf110405d90315bdb3e5d87c4, State: Running, Role: LEADER
I20260926 03:35:31.621598 16484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e2037cbf110405d90315bdb3e5d87c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43539 } }
I20260926 03:35:31.621989 16480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:35:31.624585 16485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5e2037cbf110405d90315bdb3e5d87c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e2037cbf110405d90315bdb3e5d87c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43539 } } }
I20260926 03:35:31.625119 16485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [sys.catalog]: This master's current role is: LEADER
I20260926 03:35:31.625500 16486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e2037cbf110405d90315bdb3e5d87c4. Latest consensus state: current_term: 2 leader_uuid: "5e2037cbf110405d90315bdb3e5d87c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e2037cbf110405d90315bdb3e5d87c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43539 } } }
I20260926 03:35:31.625938 16486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [sys.catalog]: This master's current role is: LEADER
I20260926 03:35:31.627719 16490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:35:31.649355 16490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:35:31.651577 16490 catalog_manager.cc:1259] Loaded cluster ID: 7d88b04aedf44fffa5e35a9554ef5ec7
I20260926 03:35:31.651798 16490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:35:31.657575 16490 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:35:31.659816 16490 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4: Loaded TSK: 0
I20260926 03:35:31.682422 16501 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 03:35:31.695888 16490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4: Generated new TSK 1
I20260926 03:35:31.696375 16490 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 03:35:31.980679 16482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:35:31.981014 16482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:35:31.981163 16482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:35:31.988965 16482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:35:31.989084 16482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:35:31.989267 16482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:35:32.000000 16482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:45301
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=34903
--tserver_master_addrs=127.0.97.62:43539
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:35:32.001941 16482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:35:32.003611 16482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:35:32.021376 16507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:35:32.021349 16508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:35:32.021507 16510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:35:32.022078 16482 server_base.cc:1034] running on GCE node
I20260926 03:35:32.023192 16482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:35:32.024591 16482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:35:32.025866 16482 hybrid_clock.cc:648] HybridClock initialized: now 1790393732025799 us; error 42 us; skew 500 ppm
I20260926 03:35:32.026350 16482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:35:32 dist-test-slave-dt7p krb5kdc[15859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:35:32.031044 16482 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:35:32.032951 16482 webserver.cc:460] Webserver started at http://127.0.97.1:34903/ using document root <none> and password file <none>
I20260926 03:35:32.033780 16482 fs_manager.cc:359] Metadata directory not provided
I20260926 03:35:32.033951 16482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:35:32.040396 16482 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.003s
I20260926 03:35:32.043594 16517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:35:32.045228 16482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:35:32.045418 16482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/wal
uuid: "274c847118904ebe8f3142653783b177"
format_stamp: "Formatted at 2026-09-26 03:35:08 on dist-test-slave-dt7p"
server_key: "a79714ed783fa6b911a7325785da2249"
server_key_iv: "7bb97d1072f73851ed64094c80cb3866"
server_key_version: "encryptionkey@0"
I20260926 03:35:32.046595 16482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:35:32.058372 16482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:35:32.062971 16482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:35:32.063268 16482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:35:32.064329 16482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:35:32.065853 16482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:35:32.065979 16482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:35:32.066115 16482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:35:32.066206 16482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:35:32.103143 16482 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45301
I20260926 03:35:32.103233 16629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45301 every 8 connection(s)
I20260926 03:35:32.104887 16482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:35:32.111521 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 16482
I20260926 03:35:32.112324 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:44503
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:43539
--webserver_port=40911
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:35:32 dist-test-slave-dt7p krb5kdc[15859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:35:32.132869 16632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:32.108193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56209 (local address 127.0.97.62:43539)
0926 03:35:32.108960 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0926 03:35:32.108983 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 03:35:32.109566 (+ 583us) server_negotiation.cc:407] Connection header received
0926 03:35:32.112201 (+ 2635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:35:32.112214 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:35:32.112677 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:35:32.113092 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:35:32.114876 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:32.116340 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:32.118081 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:32.118801 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:32.119082 (+ 281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:32.124167 (+ 5085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:35:32.124177 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:35:32.124192 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:35:32.124250 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:35:32.127691 (+ 3441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:35:32.128732 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:35:32.128740 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:35:32.128745 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:35:32.128917 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:35:32.129519 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:35:32.129523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:35:32.129525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:35:32.130226 (+ 701us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:35:32.130404 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 03:35:32.131376 (+ 972us) server_negotiation.cc:299] Negotiation successful
0926 03:35:32.131706 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":377,"threads_started":1}
I20260926 03:35:32.133157 16633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:32.108406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.1:56209)
0926 03:35:32.109366 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:32.109407 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 03:35:32.111501 (+ 2094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:35:32.113231 (+ 1730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:35:32.113241 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:35:32.113973 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:35:32.114617 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:35:32.114637 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:35:32.116519 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:35:32.116528 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:35:32.117897 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:35:32.117910 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:35:32.119060 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:35:32.119069 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:35:32.119394 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:32.120311 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:35:32.120350 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:35:32.123826 (+ 3476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:35:32.127842 (+ 4016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:35:32.127854 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:35:32.127871 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:35:32.128586 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:35:32.129037 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:35:32.129045 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:35:32.129051 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:35:32.129304 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:35:32.130371 (+ 1067us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:35:32.130383 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:35:32.130855 (+ 472us) client_negotiation.cc:769] Sending connection context
0926 03:35:32.131161 (+ 306us) client_negotiation.cc:240] Negotiation successful
0926 03:35:32.131760 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":454,"threads_started":1}
I20260926 03:35:32.136294 16630 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43539
I20260926 03:35:32.136710 16630 heartbeater.cc:464] Registering TS with master...
I20260926 03:35:32.137570 16630 heartbeater.cc:511] Master 127.0.97.62:43539 requested a full tablet report, sending...
I20260926 03:35:32.140143 16445 ts_manager.cc:194] Registered new tserver with Master: 274c847118904ebe8f3142653783b177 (127.0.97.1:45301)
I20260926 03:35:32.142549 16445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56209
W20260926 03:35:32.496388 16634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:35:32.496846 16634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:35:32.497035 16634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:35:32.504984 16634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:35:32.505157 16634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:35:32.505347 16634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:35:32.516309 16634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:44503
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=40911
--tserver_master_addrs=127.0.97.62:43539
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:35:32.518020 16634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:35:32.519560 16634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:35:32.534518 16640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:35:32.534605 16642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:35:32.534487 16639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:35:32.535782 16634 server_base.cc:1034] running on GCE node
I20260926 03:35:32.536552 16634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:35:32.537869 16634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:35:32.539088 16634 hybrid_clock.cc:648] HybridClock initialized: now 1790393732539037 us; error 42 us; skew 500 ppm
I20260926 03:35:32.539471 16634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:35:32 dist-test-slave-dt7p krb5kdc[15859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:35:32.543404 16634 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:35:32.545200 16634 webserver.cc:460] Webserver started at http://127.0.97.2:40911/ using document root <none> and password file <none>
I20260926 03:35:32.546052 16634 fs_manager.cc:359] Metadata directory not provided
I20260926 03:35:32.546222 16634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:35:32.553000 16634 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.003s
I20260926 03:35:32.556267 16649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:35:32.557907 16634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:35:32.558115 16634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/wal
uuid: "36181f43ae0844ea85f82c508cfdec95"
format_stamp: "Formatted at 2026-09-26 03:35:09 on dist-test-slave-dt7p"
server_key: "6dbeb6dedc2d5978314e0b05541991fc"
server_key_iv: "5e13ce2905cf19692b7d5f56a62c17df"
server_key_version: "encryptionkey@0"
I20260926 03:35:32.559278 16634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:35:32.588711 16634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:35:32.593443 16634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:35:32.593772 16634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:35:32.594949 16634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:35:32.596554 16634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:35:32.596747 16634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 03:35:32.596902 16634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:35:32.597018 16634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:35:32.634543 16634 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44503
I20260926 03:35:32.634596 16761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44503 every 8 connection(s)
I20260926 03:35:32.636256 16634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:35:32.644032 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 16634
I20260926 03:35:32.644857 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:40249
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:43539
--webserver_port=44143
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:35:32 dist-test-slave-dt7p krb5kdc[15859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:35:32.660782 16764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:32.639214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37517 (local address 127.0.97.62:43539)
0926 03:35:32.639727 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0926 03:35:32.639733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:35:32.640848 (+ 1115us) server_negotiation.cc:407] Connection header received
0926 03:35:32.642932 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:35:32.642938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:35:32.643062 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:35:32.643206 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:35:32.645144 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:32.645986 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:32.647718 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:32.648199 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:32.648362 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:32.653270 (+ 4908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:35:32.653277 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:35:32.653282 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:35:32.653352 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:35:32.657150 (+ 3798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:35:32.658027 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:35:32.658033 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:35:32.658037 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:35:32.658180 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:35:32.658660 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:35:32.658665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:35:32.658668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:35:32.658999 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:35:32.659190 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 03:35:32.660047 (+ 857us) server_negotiation.cc:299] Negotiation successful
0926 03:35:32.660324 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":253,"threads_started":1}
I20260926 03:35:32.662137 16765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:32.639819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.2:37517)
0926 03:35:32.640654 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:32.640701 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 03:35:32.642619 (+ 1918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:35:32.643477 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:35:32.643486 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:35:32.644089 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:35:32.644882 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:35:32.644902 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:35:32.646167 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:35:32.646172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:35:32.647551 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:35:32.647564 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:35:32.648471 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:35:32.648482 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:35:32.648798 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:32.649665 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:35:32.649744 (+ 79us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:35:32.652841 (+ 3097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:35:32.657327 (+ 4486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:35:32.657338 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:35:32.657361 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:35:32.657879 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:35:32.658302 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:35:32.658307 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:35:32.658311 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:35:32.658531 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:35:32.659148 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:35:32.659158 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:35:32.659618 (+ 460us) client_negotiation.cc:769] Sending connection context
0926 03:35:32.659952 (+ 334us) client_negotiation.cc:240] Negotiation successful
0926 03:35:32.660650 (+ 698us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":292,"threads_started":1}
I20260926 03:35:32.663988 16762 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43539
I20260926 03:35:32.664434 16762 heartbeater.cc:464] Registering TS with master...
I20260926 03:35:32.665509 16762 heartbeater.cc:511] Master 127.0.97.62:43539 requested a full tablet report, sending...
I20260926 03:35:32.667470 16445 ts_manager.cc:194] Registered new tserver with Master: 36181f43ae0844ea85f82c508cfdec95 (127.0.97.2:44503)
I20260926 03:35:32.668469 16445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37517
W20260926 03:35:33.029692 16766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:35:33.030151 16766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:35:33.030330 16766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:35:33.038365 16766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:35:33.038519 16766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:35:33.038719 16766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:35:33.049613 16766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:40249
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=44143
--tserver_master_addrs=127.0.97.62:43539
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:35:33.051308 16766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:35:33.052977 16766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:35:33.067639 16771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:35:33.067687 16772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:35:33.067719 16774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:35:33.068842 16766 server_base.cc:1034] running on GCE node
I20260926 03:35:33.069651 16766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:35:33.070698 16766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:35:33.071890 16766 hybrid_clock.cc:648] HybridClock initialized: now 1790393733071831 us; error 48 us; skew 500 ppm
I20260926 03:35:33.072225 16766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:35:33 dist-test-slave-dt7p krb5kdc[15859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:35:33.075691 16766 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:35:33.077551 16766 webserver.cc:460] Webserver started at http://127.0.97.3:44143/ using document root <none> and password file <none>
I20260926 03:35:33.078339 16766 fs_manager.cc:359] Metadata directory not provided
I20260926 03:35:33.078506 16766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:35:33.084547 16766 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 03:35:33.087510 16781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:35:33.089032 16766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 03:35:33.089213 16766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/wal
uuid: "157e59415ba045a5815168d80ffa59b1"
format_stamp: "Formatted at 2026-09-26 03:35:10 on dist-test-slave-dt7p"
server_key: "52c4d483c830ebdffc58ec3f2df12a18"
server_key_iv: "064ec938c8c2670e097170ab00301d57"
server_key_version: "encryptionkey@0"
I20260926 03:35:33.090305 16766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:35:33.102571 16766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:35:33.107223 16766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:35:33.107542 16766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:35:33.108639 16766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:35:33.110036 16766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:35:33.110157 16766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:35:33.110301 16766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:35:33.110389 16766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:35:33.146241 16630 heartbeater.cc:503] Master 127.0.97.62:43539 was elected leader, sending a full tablet report...
I20260926 03:35:33.147248 16766 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40249
I20260926 03:35:33.147289 16893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40249 every 8 connection(s)
I20260926 03:35:33.149042 16766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:35:33.157183 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 16766
Sep 26 03:35:33 dist-test-slave-dt7p krb5kdc[15859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:35:33.171039 16764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:33.151885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56395 (local address 127.0.97.62:43539)
0926 03:35:33.152116 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 03:35:33.152123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:35:33.153507 (+ 1384us) server_negotiation.cc:407] Connection header received
0926 03:35:33.155394 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:35:33.155398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:35:33.155473 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:35:33.155559 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:35:33.157220 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:33.158232 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:33.160095 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:33.160534 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:33.160702 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:33.164798 (+ 4096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:35:33.164803 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:35:33.164806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:35:33.164845 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:35:33.167835 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:35:33.168621 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:35:33.168626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:35:33.168628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:35:33.168748 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:35:33.169214 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:35:33.169219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:35:33.169233 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:35:33.169554 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:35:33.169752 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 03:35:33.170443 (+ 691us) server_negotiation.cc:299] Negotiation successful
0926 03:35:33.170670 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 03:35:33.172120 16896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:33.152511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.3:56395)
0926 03:35:33.153338 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:33.153388 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 03:35:33.155122 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:35:33.155772 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:35:33.155780 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:35:33.156345 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:35:33.157045 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:35:33.157062 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:35:33.158471 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:35:33.158481 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:35:33.159952 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:35:33.159964 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:35:33.160805 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:35:33.160813 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:35:33.161050 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:33.161811 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:35:33.161834 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:35:33.164617 (+ 2783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:35:33.167993 (+ 3376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:35:33.168002 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:35:33.168018 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:35:33.168451 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:35:33.168901 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:35:33.168906 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:35:33.168910 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:35:33.169095 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:35:33.169700 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:35:33.169710 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:35:33.170075 (+ 365us) client_negotiation.cc:769] Sending connection context
0926 03:35:33.170355 (+ 280us) client_negotiation.cc:240] Negotiation successful
0926 03:35:33.170902 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":304,"threads_started":1}
I20260926 03:35:33.173334 16897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:33.158173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53644 (local address 127.0.97.62:43539)
0926 03:35:33.158793 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0926 03:35:33.158800 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:35:33.159099 (+ 299us) server_negotiation.cc:407] Connection header received
0926 03:35:33.159305 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:35:33.159312 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:35:33.159372 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:35:33.159484 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:35:33.161207 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:33.162158 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:33.163978 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:33.164408 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:33.164499 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:33.167434 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:35:33.167439 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:35:33.167442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:35:33.167485 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:35:33.170260 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:35:33.170971 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:35:33.170979 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:35:33.170987 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:35:33.171126 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:35:33.171567 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:35:33.171573 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:35:33.171577 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:35:33.171855 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:35:33.172139 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0926 03:35:33.172761 (+ 622us) server_negotiation.cc:299] Negotiation successful
0926 03:35:33.172972 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":343,"threads_started":1}
I20260926 03:35:33.173856 16894 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43539
I20260926 03:35:33.174270 16894 heartbeater.cc:464] Registering TS with master...
I20260926 03:35:33.175099 16894 heartbeater.cc:511] Master 127.0.97.62:43539 requested a full tablet report, sending...
I20260926 03:35:33.176738 16445 ts_manager.cc:194] Registered new tserver with Master: 157e59415ba045a5815168d80ffa59b1 (127.0.97.3:40249)
I20260926 03:35:33.177630 16445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:56395
I20260926 03:35:33.188189 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:35:33.195055 16897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:33.189344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53648 (local address 127.0.97.62:43539)
0926 03:35:33.189609 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 03:35:33.189618 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:35:33.189917 (+ 299us) server_negotiation.cc:407] Connection header received
0926 03:35:33.190142 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:35:33.190146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:35:33.190336 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:35:33.190472 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:35:33.190859 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:33.191599 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:33.193086 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:33.193564 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:33.193661 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:33.194110 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:35:33.194387 (+ 277us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 03:35:33.194760 (+ 373us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53648: token expired
Metrics: {"server-negotiator.queue_time_us":77}
W20260926 03:35:33.195191 16897 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53648: token expired
W20260926 03:35:33.195777 16367 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:43539)
I20260926 03:35:33.209633 16897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:33.196267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53656 (local address 127.0.97.62:43539)
0926 03:35:33.196437 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 03:35:33.196443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:35:33.196485 (+ 42us) server_negotiation.cc:407] Connection header received
0926 03:35:33.196721 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:35:33.196726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:35:33.196785 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:35:33.196886 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:35:33.198292 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:33.198991 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:33.200373 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:33.200849 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:33.200995 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:33.203921 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:35:33.203926 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:35:33.203930 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:35:33.203965 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:35:33.206724 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:35:33.207445 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:35:33.207452 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:35:33.207455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:35:33.207598 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:35:33.207986 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:35:33.207993 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:35:33.207996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:35:33.208349 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:35:33.208681 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0926 03:35:33.208976 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 03:35:33.209273 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 03:35:33.213568 16445 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:35:33.222150 16445 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:35:33.251648 16905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:33.236955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40249 (local address 127.0.0.1:35418)
0926 03:35:33.242665 (+ 5710us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:33.242686 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:35:33.242829 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:35:33.243862 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:35:33.243866 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:35:33.243890 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:35:33.244404 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:35:33.244420 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:35:33.246217 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:35:33.246222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:35:33.248360 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:35:33.248378 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:35:33.250547 (+ 2169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:35:33.250551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:35:33.250690 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:33.250725 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 03:35:33.250878 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 03:35:33.251079 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5582,"mutex_wait_us":120,"thread_start_us":5427,"threads_started":1}
I20260926 03:35:33.251765 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:33.237906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35418 (local address 127.0.97.3:40249)
0926 03:35:33.241247 (+ 3341us) server_negotiation.cc:206] Beginning negotiation
0926 03:35:33.241269 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 03:35:33.242998 (+ 1729us) server_negotiation.cc:407] Connection header received
0926 03:35:33.243116 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:35:33.243124 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:35:33.243344 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:35:33.243530 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:35:33.244666 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:33.246044 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:33.248524 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:33.249974 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:33.250105 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:33.250405 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0926 03:35:33.250948 (+ 543us) server_negotiation.cc:299] Negotiation successful
0926 03:35:33.251125 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3103,"thread_start_us":2888,"threads_started":1}
I20260926 03:35:33.252506 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:33.238659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59146 (local address 127.0.97.2:44503)
0926 03:35:33.239317 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0926 03:35:33.239326 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:35:33.243286 (+ 3960us) server_negotiation.cc:407] Connection header received
0926 03:35:33.243631 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:35:33.243639 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:35:33.243813 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:35:33.244008 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:35:33.244762 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:33.246400 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:33.249225 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:33.250781 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:33.250872 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:33.251070 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 03:35:33.251970 (+ 900us) server_negotiation.cc:299] Negotiation successful
0926 03:35:33.252162 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":562,"threads_started":1}
I20260926 03:35:33.252524 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:33.237031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56276 (local address 127.0.97.1:45301)
0926 03:35:33.238618 (+ 1587us) server_negotiation.cc:206] Beginning negotiation
0926 03:35:33.238629 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:35:33.242048 (+ 3419us) server_negotiation.cc:407] Connection header received
0926 03:35:33.242570 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:35:33.242581 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:35:33.242901 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:35:33.243081 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:35:33.244225 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:33.246069 (+ 1844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:33.249224 (+ 3155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:33.250759 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:33.250868 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:33.251097 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 03:35:33.251976 (+ 879us) server_negotiation.cc:299] Negotiation successful
0926 03:35:33.252154 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1317,"thread_start_us":1815,"threads_started":1}
I20260926 03:35:33.253614 16909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:33.236738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45301 (local address 127.0.0.1:56276)
0926 03:35:33.241890 (+ 5152us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:33.241932 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 03:35:33.242223 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:35:33.243467 (+ 1244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:35:33.243479 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:35:33.243512 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:35:33.243987 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:35:33.244007 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:35:33.246248 (+ 2241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:35:33.246293 (+ 45us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:35:33.249033 (+ 2740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:35:33.249048 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:35:33.251335 (+ 2287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:35:33.251346 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:35:33.251672 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:33.251788 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 03:35:33.252830 (+ 1042us) client_negotiation.cc:240] Negotiation successful
0926 03:35:33.253076 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4886,"mutex_wait_us":262,"thread_start_us":6110,"threads_started":1}
I20260926 03:35:33.253643 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:33.237664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44503 (local address 127.0.0.1:59146)
0926 03:35:33.243161 (+ 5497us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:33.243178 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:35:33.243395 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:35:33.243999 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:35:33.244008 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:35:33.244034 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:35:33.244478 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:35:33.244488 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:35:33.246579 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:35:33.246590 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:35:33.249033 (+ 2443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:35:33.249048 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:35:33.251366 (+ 2318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:35:33.251371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:35:33.251668 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:33.251791 (+ 123us) client_negotiation.cc:769] Sending connection context
0926 03:35:33.252837 (+ 1046us) client_negotiation.cc:240] Negotiation successful
0926 03:35:33.253076 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5379,"mutex_wait_us":185,"thread_start_us":3825,"threads_started":1}
I20260926 03:35:33.257175 16829 tablet_service.cc:1467] Processing CreateTablet for tablet f5ffbef3a4a44165b5b8e061b0460d9e (DEFAULT_TABLE table=test-table [id=6e41781b8c3b43f083b6ff3283113731]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:35:33.257396 16697 tablet_service.cc:1467] Processing CreateTablet for tablet f5ffbef3a4a44165b5b8e061b0460d9e (DEFAULT_TABLE table=test-table [id=6e41781b8c3b43f083b6ff3283113731]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:35:33.258952 16697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5ffbef3a4a44165b5b8e061b0460d9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:35:33.258955 16829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5ffbef3a4a44165b5b8e061b0460d9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:35:33.258754 16565 tablet_service.cc:1467] Processing CreateTablet for tablet f5ffbef3a4a44165b5b8e061b0460d9e (DEFAULT_TABLE table=test-table [id=6e41781b8c3b43f083b6ff3283113731]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:35:33.260228 16565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5ffbef3a4a44165b5b8e061b0460d9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:35:33.274725 16910 tablet_bootstrap.cc:492] T f5ffbef3a4a44165b5b8e061b0460d9e P 274c847118904ebe8f3142653783b177: Bootstrap starting.
I20260926 03:35:33.278769 16910 tablet_bootstrap.cc:654] T f5ffbef3a4a44165b5b8e061b0460d9e P 274c847118904ebe8f3142653783b177: Neither blocks nor log segments found. Creating new log.
I20260926 03:35:33.280122 16910 log.cc:824] T f5ffbef3a4a44165b5b8e061b0460d9e P 274c847118904ebe8f3142653783b177: Log is configured to *not* fsync() on all Append() calls
I20260926 03:35:33.281539 16911 tablet_bootstrap.cc:492] T f5ffbef3a4a44165b5b8e061b0460d9e P 36181f43ae0844ea85f82c508cfdec95: Bootstrap starting.
I20260926 03:35:33.283983 16912 tablet_bootstrap.cc:492] T f5ffbef3a4a44165b5b8e061b0460d9e P 157e59415ba045a5815168d80ffa59b1: Bootstrap starting.
I20260926 03:35:33.286562 16910 tablet_bootstrap.cc:492] T f5ffbef3a4a44165b5b8e061b0460d9e P 274c847118904ebe8f3142653783b177: No bootstrap required, opened a new log
I20260926 03:35:33.286892 16910 ts_tablet_manager.cc:1397] T f5ffbef3a4a44165b5b8e061b0460d9e P 274c847118904ebe8f3142653783b177: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260926 03:35:33.288225 16911 tablet_bootstrap.cc:654] T f5ffbef3a4a44165b5b8e061b0460d9e P 36181f43ae0844ea85f82c508cfdec95: Neither blocks nor log segments found. Creating new log.
I20260926 03:35:33.288223 16912 tablet_bootstrap.cc:654] T f5ffbef3a4a44165b5b8e061b0460d9e P 157e59415ba045a5815168d80ffa59b1: Neither blocks nor log segments found. Creating new log.
I20260926 03:35:33.289692 16911 log.cc:824] T f5ffbef3a4a44165b5b8e061b0460d9e P 36181f43ae0844ea85f82c508cfdec95: Log is configured to *not* fsync() on all Append() calls
I20260926 03:35:33.289693 16912 log.cc:824] T f5ffbef3a4a44165b5b8e061b0460d9e P 157e59415ba045a5815168d80ffa59b1: Log is configured to *not* fsync() on all Append() calls
I20260926 03:35:33.293155 16912 tablet_bootstrap.cc:492] T f5ffbef3a4a44165b5b8e061b0460d9e P 157e59415ba045a5815168d80ffa59b1: No bootstrap required, opened a new log
I20260926 03:35:33.293864 16911 tablet_bootstrap.cc:492] T f5ffbef3a4a44165b5b8e061b0460d9e P 36181f43ae0844ea85f82c508cfdec95: No bootstrap required, opened a new log
I20260926 03:35:33.294005 16912 ts_tablet_manager.cc:1397] T f5ffbef3a4a44165b5b8e061b0460d9e P 157e59415ba045a5815168d80ffa59b1: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.004s
I20260926 03:35:33.294257 16911 ts_tablet_manager.cc:1397] T f5ffbef3a4a44165b5b8e061b0460d9e P 36181f43ae0844ea85f82c508cfdec95: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20260926 03:35:33.296209 16910 raft_consensus.cc:348] T f5ffbef3a4a44165b5b8e061b0460d9e P 274c847118904ebe8f3142653783b177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274c847118904ebe8f3142653783b177" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45301 } } peers { permanent_uuid: "157e59415ba045a5815168d80ffa59b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40249 } } peers { permanent_uuid: "36181f43ae0844ea85f82c508cfdec95" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44503 } }
I20260926 03:35:33.296661 16910 raft_consensus.cc:374] T f5ffbef3a4a44165b5b8e061b0460d9e P 274c847118904ebe8f3142653783b177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:35:33.296820 16910 raft_consensus.cc:729] T f5ffbef3a4a44165b5b8e061b0460d9e P 274c847118904ebe8f3142653783b177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 274c847118904ebe8f3142653783b177, State: Initialized, Role: FOLLOWER
I20260926 03:35:33.297489 16910 consensus_queue.cc:260] T f5ffbef3a4a44165b5b8e061b0460d9e P 274c847118904ebe8f3142653783b177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274c847118904ebe8f3142653783b177" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45301 } } peers { permanent_uuid: "157e59415ba045a5815168d80ffa59b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40249 } } peers { permanent_uuid: "36181f43ae0844ea85f82c508cfdec95" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44503 } }
I20260926 03:35:33.299222 16910 ts_tablet_manager.cc:1428] T f5ffbef3a4a44165b5b8e061b0460d9e P 274c847118904ebe8f3142653783b177: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260926 03:35:33.305032 16911 raft_consensus.cc:348] T f5ffbef3a4a44165b5b8e061b0460d9e P 36181f43ae0844ea85f82c508cfdec95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274c847118904ebe8f3142653783b177" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45301 } } peers { permanent_uuid: "157e59415ba045a5815168d80ffa59b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40249 } } peers { permanent_uuid: "36181f43ae0844ea85f82c508cfdec95" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44503 } }
I20260926 03:35:33.305570 16911 raft_consensus.cc:374] T f5ffbef3a4a44165b5b8e061b0460d9e P 36181f43ae0844ea85f82c508cfdec95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:35:33.305677 16911 raft_consensus.cc:729] T f5ffbef3a4a44165b5b8e061b0460d9e P 36181f43ae0844ea85f82c508cfdec95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36181f43ae0844ea85f82c508cfdec95, State: Initialized, Role: FOLLOWER
I20260926 03:35:33.306367 16911 consensus_queue.cc:260] T f5ffbef3a4a44165b5b8e061b0460d9e P 36181f43ae0844ea85f82c508cfdec95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274c847118904ebe8f3142653783b177" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45301 } } peers { permanent_uuid: "157e59415ba045a5815168d80ffa59b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40249 } } peers { permanent_uuid: "36181f43ae0844ea85f82c508cfdec95" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44503 } }
I20260926 03:35:33.309283 16762 heartbeater.cc:503] Master 127.0.97.62:43539 was elected leader, sending a full tablet report...
I20260926 03:35:33.310091 16911 ts_tablet_manager.cc:1428] T f5ffbef3a4a44165b5b8e061b0460d9e P 36181f43ae0844ea85f82c508cfdec95: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
I20260926 03:35:33.310226 16912 raft_consensus.cc:348] T f5ffbef3a4a44165b5b8e061b0460d9e P 157e59415ba045a5815168d80ffa59b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274c847118904ebe8f3142653783b177" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45301 } } peers { permanent_uuid: "157e59415ba045a5815168d80ffa59b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40249 } } peers { permanent_uuid: "36181f43ae0844ea85f82c508cfdec95" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44503 } }
I20260926 03:35:33.310674 16912 raft_consensus.cc:374] T f5ffbef3a4a44165b5b8e061b0460d9e P 157e59415ba045a5815168d80ffa59b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:35:33.310817 16912 raft_consensus.cc:729] T f5ffbef3a4a44165b5b8e061b0460d9e P 157e59415ba045a5815168d80ffa59b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 157e59415ba045a5815168d80ffa59b1, State: Initialized, Role: FOLLOWER
I20260926 03:35:33.311393 16912 consensus_queue.cc:260] T f5ffbef3a4a44165b5b8e061b0460d9e P 157e59415ba045a5815168d80ffa59b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274c847118904ebe8f3142653783b177" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45301 } } peers { permanent_uuid: "157e59415ba045a5815168d80ffa59b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40249 } } peers { permanent_uuid: "36181f43ae0844ea85f82c508cfdec95" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44503 } }
I20260926 03:35:33.312539 16894 heartbeater.cc:503] Master 127.0.97.62:43539 was elected leader, sending a full tablet report...
I20260926 03:35:33.314162 16912 ts_tablet_manager.cc:1428] T f5ffbef3a4a44165b5b8e061b0460d9e P 157e59415ba045a5815168d80ffa59b1: Time spent starting tablet: real 0.020s user 0.014s sys 0.000s
W20260926 03:35:33.358116 16631 tablet.cc:2369] T f5ffbef3a4a44165b5b8e061b0460d9e P 274c847118904ebe8f3142653783b177: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:35:33.388764 16763 tablet.cc:2369] T f5ffbef3a4a44165b5b8e061b0460d9e P 36181f43ae0844ea85f82c508cfdec95: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:35:33.395901 16918 raft_consensus.cc:482] T f5ffbef3a4a44165b5b8e061b0460d9e P 157e59415ba045a5815168d80ffa59b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:35:33.396327 16918 raft_consensus.cc:504] T f5ffbef3a4a44165b5b8e061b0460d9e P 157e59415ba045a5815168d80ffa59b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274c847118904ebe8f3142653783b177" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45301 } } peers { permanent_uuid: "157e59415ba045a5815168d80ffa59b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40249 } } peers { permanent_uuid: "36181f43ae0844ea85f82c508cfdec95" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44503 } }
I20260926 03:35:33.398703 16918 leader_election.cc:288] T f5ffbef3a4a44165b5b8e061b0460d9e P 157e59415ba045a5815168d80ffa59b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 274c847118904ebe8f3142653783b177 (127.0.97.1:45301), 36181f43ae0844ea85f82c508cfdec95 (127.0.97.2:44503)
W20260926 03:35:33.400805 16895 tablet.cc:2369] T f5ffbef3a4a44165b5b8e061b0460d9e P 157e59415ba045a5815168d80ffa59b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:35:33.406029 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:33.399111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39127 (local address 127.0.97.1:45301)
0926 03:35:33.399362 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 03:35:33.399369 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:35:33.399719 (+ 350us) server_negotiation.cc:407] Connection header received
0926 03:35:33.399845 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:35:33.399851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:35:33.399951 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:35:33.400051 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:35:33.400553 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:33.401753 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:33.403848 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:33.405115 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:33.405214 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:33.405301 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 03:35:33.405674 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 03:35:33.405791 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 03:35:33.406106 16896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:33.398955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45301 (local address 127.0.97.3:39127)
0926 03:35:33.399343 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:33.399373 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 03:35:33.399537 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:35:33.400116 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:35:33.400124 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:35:33.400157 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:35:33.400405 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:35:33.400415 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:35:33.401875 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:35:33.401879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:35:33.403660 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:35:33.403672 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:35:33.405261 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:35:33.405266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:35:33.405469 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:33.405507 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 03:35:33.405612 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 03:35:33.405770 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":36,"spinlock_wait_cycles":1280}
I20260926 03:35:33.407204 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:33.399390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54275 (local address 127.0.97.2:44503)
0926 03:35:33.399655 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 03:35:33.399665 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:35:33.400179 (+ 514us) server_negotiation.cc:407] Connection header received
0926 03:35:33.400351 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:35:33.400355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:35:33.400420 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:35:33.400528 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:35:33.401698 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:33.402802 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:33.404990 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:33.406211 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:33.406345 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:33.406462 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 03:35:33.406890 (+ 428us) server_negotiation.cc:299] Negotiation successful
0926 03:35:33.406996 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 03:35:33.407481 16919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:33.399216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44503 (local address 127.0.97.3:54275)
0926 03:35:33.400052 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:33.400077 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 03:35:33.400194 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:35:33.401014 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:35:33.401019 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:35:33.401044 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:35:33.401497 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:35:33.401509 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:35:33.402948 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:35:33.402951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:35:33.404838 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:35:33.404855 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:35:33.406527 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:35:33.406539 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:35:33.406708 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:33.406746 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 03:35:33.406871 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 03:35:33.407007 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"mutex_wait_us":1,"thread_start_us":518,"threads_started":1}
I20260926 03:35:33.407364 16585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5ffbef3a4a44165b5b8e061b0460d9e" candidate_uuid: "157e59415ba045a5815168d80ffa59b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274c847118904ebe8f3142653783b177" is_pre_election: true
I20260926 03:35:33.408185 16585 raft_consensus.cc:2454] T f5ffbef3a4a44165b5b8e061b0460d9e P 274c847118904ebe8f3142653783b177 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 157e59415ba045a5815168d80ffa59b1 in term 0.
I20260926 03:35:33.408712 16717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5ffbef3a4a44165b5b8e061b0460d9e" candidate_uuid: "157e59415ba045a5815168d80ffa59b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36181f43ae0844ea85f82c508cfdec95" is_pre_election: true
I20260926 03:35:33.409112 16785 leader_election.cc:302] T f5ffbef3a4a44165b5b8e061b0460d9e P 157e59415ba045a5815168d80ffa59b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 157e59415ba045a5815168d80ffa59b1, 274c847118904ebe8f3142653783b177; no voters:
I20260926 03:35:33.409317 16717 raft_consensus.cc:2454] T f5ffbef3a4a44165b5b8e061b0460d9e P 36181f43ae0844ea85f82c508cfdec95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 157e59415ba045a5815168d80ffa59b1 in term 0.
I20260926 03:35:33.409507 16918 raft_consensus.cc:2781] T f5ffbef3a4a44165b5b8e061b0460d9e P 157e59415ba045a5815168d80ffa59b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:35:33.409662 16918 raft_consensus.cc:482] T f5ffbef3a4a44165b5b8e061b0460d9e P 157e59415ba045a5815168d80ffa59b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:35:33.409781 16918 raft_consensus.cc:3037] T f5ffbef3a4a44165b5b8e061b0460d9e P 157e59415ba045a5815168d80ffa59b1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:35:33.412381 16918 raft_consensus.cc:504] T f5ffbef3a4a44165b5b8e061b0460d9e P 157e59415ba045a5815168d80ffa59b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274c847118904ebe8f3142653783b177" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45301 } } peers { permanent_uuid: "157e59415ba045a5815168d80ffa59b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40249 } } peers { permanent_uuid: "36181f43ae0844ea85f82c508cfdec95" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44503 } }
I20260926 03:35:33.413103 16918 leader_election.cc:288] T f5ffbef3a4a44165b5b8e061b0460d9e P 157e59415ba045a5815168d80ffa59b1 [CANDIDATE]: Term 1 election: Requested vote from peers 274c847118904ebe8f3142653783b177 (127.0.97.1:45301), 36181f43ae0844ea85f82c508cfdec95 (127.0.97.2:44503)
I20260926 03:35:33.413543 16717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5ffbef3a4a44165b5b8e061b0460d9e" candidate_uuid: "157e59415ba045a5815168d80ffa59b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36181f43ae0844ea85f82c508cfdec95"
I20260926 03:35:33.413587 16585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5ffbef3a4a44165b5b8e061b0460d9e" candidate_uuid: "157e59415ba045a5815168d80ffa59b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274c847118904ebe8f3142653783b177"
I20260926 03:35:33.413760 16717 raft_consensus.cc:3037] T f5ffbef3a4a44165b5b8e061b0460d9e P 36181f43ae0844ea85f82c508cfdec95 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:35:33.413820 16585 raft_consensus.cc:3037] T f5ffbef3a4a44165b5b8e061b0460d9e P 274c847118904ebe8f3142653783b177 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:35:33.416239 16585 raft_consensus.cc:2454] T f5ffbef3a4a44165b5b8e061b0460d9e P 274c847118904ebe8f3142653783b177 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 157e59415ba045a5815168d80ffa59b1 in term 1.
I20260926 03:35:33.416271 16717 raft_consensus.cc:2454] T f5ffbef3a4a44165b5b8e061b0460d9e P 36181f43ae0844ea85f82c508cfdec95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 157e59415ba045a5815168d80ffa59b1 in term 1.
I20260926 03:35:33.416648 16785 leader_election.cc:302] T f5ffbef3a4a44165b5b8e061b0460d9e P 157e59415ba045a5815168d80ffa59b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 157e59415ba045a5815168d80ffa59b1, 274c847118904ebe8f3142653783b177; no voters:
I20260926 03:35:33.416934 16918 raft_consensus.cc:2781] T f5ffbef3a4a44165b5b8e061b0460d9e P 157e59415ba045a5815168d80ffa59b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:35:33.417374 16918 raft_consensus.cc:686] T f5ffbef3a4a44165b5b8e061b0460d9e P 157e59415ba045a5815168d80ffa59b1 [term 1 LEADER]: Becoming Leader. State: Replica: 157e59415ba045a5815168d80ffa59b1, State: Running, Role: LEADER
I20260926 03:35:33.417888 16918 consensus_queue.cc:237] T f5ffbef3a4a44165b5b8e061b0460d9e P 157e59415ba045a5815168d80ffa59b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274c847118904ebe8f3142653783b177" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45301 } } peers { permanent_uuid: "157e59415ba045a5815168d80ffa59b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40249 } } peers { permanent_uuid: "36181f43ae0844ea85f82c508cfdec95" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44503 } }
I20260926 03:35:33.423372 16445 catalog_manager.cc:5495] T f5ffbef3a4a44165b5b8e061b0460d9e P 157e59415ba045a5815168d80ffa59b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 157e59415ba045a5815168d80ffa59b1 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "157e59415ba045a5815168d80ffa59b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "274c847118904ebe8f3142653783b177" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "157e59415ba045a5815168d80ffa59b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36181f43ae0844ea85f82c508cfdec95" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44503 } health_report { overall_health: UNKNOWN } } }
I20260926 03:35:33.446576 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:33.441158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35426 (local address 127.0.97.3:40249)
0926 03:35:33.441373 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 03:35:33.441377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:35:33.441442 (+ 65us) server_negotiation.cc:407] Connection header received
0926 03:35:33.441637 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:35:33.441641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:35:33.441731 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:35:33.441872 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:35:33.442218 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:33.442974 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:33.444298 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:33.444783 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:33.444901 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:33.445382 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:35:33.445990 (+ 608us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:35:33.446109 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 03:35:33.446173 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 03:35:33.446248 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 03:35:33.461014 16585 raft_consensus.cc:1261] T f5ffbef3a4a44165b5b8e061b0460d9e P 274c847118904ebe8f3142653783b177 [term 1 FOLLOWER]: Refusing update from remote peer 157e59415ba045a5815168d80ffa59b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:35:33.461323 16717 raft_consensus.cc:1261] T f5ffbef3a4a44165b5b8e061b0460d9e P 36181f43ae0844ea85f82c508cfdec95 [term 1 FOLLOWER]: Refusing update from remote peer 157e59415ba045a5815168d80ffa59b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:35:33.462378 16920 consensus_queue.cc:1035] T f5ffbef3a4a44165b5b8e061b0460d9e P 157e59415ba045a5815168d80ffa59b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "274c847118904ebe8f3142653783b177" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:35:33.465003 16920 consensus_queue.cc:1035] T f5ffbef3a4a44165b5b8e061b0460d9e P 157e59415ba045a5815168d80ffa59b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36181f43ae0844ea85f82c508cfdec95" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:35:33.476487 16922 mvcc.cc:205] Tried to move back new op lower bound from 7333452732240125952 to 7333452732083712000. Current Snapshot: MvccSnapshot[applied={T|T < 7333452732240125952}]
I20260926 03:35:33.495671 16444 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:35:33.496125 16444 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:35:33.501538 16444 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4: Sending DeleteTablet for 3 replicas of tablet f5ffbef3a4a44165b5b8e061b0460d9e
I20260926 03:35:33.503219 16565 tablet_service.cc:1514] Processing DeleteTablet for tablet f5ffbef3a4a44165b5b8e061b0460d9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:35:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56276
I20260926 03:35:33.503391 16829 tablet_service.cc:1514] Processing DeleteTablet for tablet f5ffbef3a4a44165b5b8e061b0460d9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:35:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35418
I20260926 03:35:33.504326 16932 tablet_replica.cc:330] T f5ffbef3a4a44165b5b8e061b0460d9e P 274c847118904ebe8f3142653783b177: stopping tablet replica
I20260926 03:35:33.504374 16697 tablet_service.cc:1514] Processing DeleteTablet for tablet f5ffbef3a4a44165b5b8e061b0460d9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:35:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59146
I20260926 03:35:33.504923 16932 raft_consensus.cc:2229] T f5ffbef3a4a44165b5b8e061b0460d9e P 274c847118904ebe8f3142653783b177 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:35:33.505601 16932 raft_consensus.cc:2258] T f5ffbef3a4a44165b5b8e061b0460d9e P 274c847118904ebe8f3142653783b177 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:35:33.506664 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 16482
I20260926 03:35:33.508291 16932 ts_tablet_manager.cc:1905] T f5ffbef3a4a44165b5b8e061b0460d9e P 274c847118904ebe8f3142653783b177: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:35:33.516974 16932 ts_tablet_manager.cc:1918] T f5ffbef3a4a44165b5b8e061b0460d9e P 274c847118904ebe8f3142653783b177: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:35:33.517351 16932 log.cc:1196] T f5ffbef3a4a44165b5b8e061b0460d9e P 274c847118904ebe8f3142653783b177: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/wal/wals/f5ffbef3a4a44165b5b8e061b0460d9e
I20260926 03:35:33.518110 16932 ts_tablet_manager.cc:1939] T f5ffbef3a4a44165b5b8e061b0460d9e P 274c847118904ebe8f3142653783b177: Deleting consensus metadata
I20260926 03:35:33.520810 16432 catalog_manager.cc:4841] TS 274c847118904ebe8f3142653783b177 (127.0.97.1:45301): tablet f5ffbef3a4a44165b5b8e061b0460d9e (table test-table [id=6e41781b8c3b43f083b6ff3283113731]) successfully deleted
I20260926 03:35:33.529229 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:33.508233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56286 (local address 127.0.97.1:45301)
0926 03:35:33.508434 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 03:35:33.508441 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:35:33.508461 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:35:33.508530 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:35:33.508535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:35:33.508674 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:35:33.508785 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:35:33.510871 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:33.512131 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:33.514549 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:33.515280 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:33.515415 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:33.519503 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:35:33.519516 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:35:33.519538 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:35:33.519585 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:35:33.524271 (+ 4686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:35:33.525373 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:35:33.525386 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:35:33.525393 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:35:33.525598 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:35:33.526138 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:35:33.526143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:35:33.526147 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:35:33.526721 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:35:33.527096 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0926 03:35:33.528500 (+ 1404us) server_negotiation.cc:299] Negotiation successful
0926 03:35:33.528803 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 03:35:33.530184 16625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:35:33.531824 16934 tablet_replica.cc:330] T f5ffbef3a4a44165b5b8e061b0460d9e P 36181f43ae0844ea85f82c508cfdec95: stopping tablet replica
I20260926 03:35:33.532672 16934 raft_consensus.cc:2229] T f5ffbef3a4a44165b5b8e061b0460d9e P 36181f43ae0844ea85f82c508cfdec95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:35:33.533458 16934 raft_consensus.cc:2258] T f5ffbef3a4a44165b5b8e061b0460d9e P 36181f43ae0844ea85f82c508cfdec95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:35:33.534174 16933 tablet_replica.cc:330] T f5ffbef3a4a44165b5b8e061b0460d9e P 157e59415ba045a5815168d80ffa59b1: stopping tablet replica
I20260926 03:35:33.535211 16933 raft_consensus.cc:2229] T f5ffbef3a4a44165b5b8e061b0460d9e P 157e59415ba045a5815168d80ffa59b1 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:35:33.536401 16933 raft_consensus.cc:2258] T f5ffbef3a4a44165b5b8e061b0460d9e P 157e59415ba045a5815168d80ffa59b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:35:33.538961 16934 ts_tablet_manager.cc:1905] T f5ffbef3a4a44165b5b8e061b0460d9e P 36181f43ae0844ea85f82c508cfdec95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:35:33.540444 16933 ts_tablet_manager.cc:1905] T f5ffbef3a4a44165b5b8e061b0460d9e P 157e59415ba045a5815168d80ffa59b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:35:33.552882 16933 ts_tablet_manager.cc:1918] T f5ffbef3a4a44165b5b8e061b0460d9e P 157e59415ba045a5815168d80ffa59b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:35:33.556995 16934 ts_tablet_manager.cc:1918] T f5ffbef3a4a44165b5b8e061b0460d9e P 36181f43ae0844ea85f82c508cfdec95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:35:33.557291 16934 log.cc:1196] T f5ffbef3a4a44165b5b8e061b0460d9e P 36181f43ae0844ea85f82c508cfdec95: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/wal/wals/f5ffbef3a4a44165b5b8e061b0460d9e
I20260926 03:35:33.557950 16934 ts_tablet_manager.cc:1939] T f5ffbef3a4a44165b5b8e061b0460d9e P 36181f43ae0844ea85f82c508cfdec95: Deleting consensus metadata
I20260926 03:35:33.558149 16933 log.cc:1196] T f5ffbef3a4a44165b5b8e061b0460d9e P 157e59415ba045a5815168d80ffa59b1: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/wal/wals/f5ffbef3a4a44165b5b8e061b0460d9e
I20260926 03:35:33.558748 16933 ts_tablet_manager.cc:1939] T f5ffbef3a4a44165b5b8e061b0460d9e P 157e59415ba045a5815168d80ffa59b1: Deleting consensus metadata
I20260926 03:35:33.561175 16430 catalog_manager.cc:4841] TS 36181f43ae0844ea85f82c508cfdec95 (127.0.97.2:44503): tablet f5ffbef3a4a44165b5b8e061b0460d9e (table test-table [id=6e41781b8c3b43f083b6ff3283113731]) successfully deleted
I20260926 03:35:33.563532 16429 catalog_manager.cc:4841] TS 157e59415ba045a5815168d80ffa59b1 (127.0.97.3:40249): tablet f5ffbef3a4a44165b5b8e061b0460d9e (table test-table [id=6e41781b8c3b43f083b6ff3283113731]) successfully deleted
I20260926 03:35:33.656019 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 16482
I20260926 03:35:33.679124 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 16634
I20260926 03:35:33.701387 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:33.680240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59156 (local address 127.0.97.2:44503)
0926 03:35:33.680491 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 03:35:33.680496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:35:33.680518 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:35:33.680631 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:35:33.680637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:35:33.680724 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:35:33.680840 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:35:33.682756 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:33.683606 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:33.685637 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:33.686060 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:33.686158 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:33.695206 (+ 9048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:35:33.695224 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:35:33.695246 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:35:33.695283 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:35:33.698508 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:35:33.699319 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:35:33.699327 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:35:33.699329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:35:33.699451 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:35:33.699833 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:35:33.699836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:35:33.699838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:35:33.700232 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:35:33.700473 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0926 03:35:33.700849 (+ 376us) server_negotiation.cc:299] Negotiation successful
0926 03:35:33.701031 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260926 03:35:33.702203 16757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:35:33.801633 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 16634
I20260926 03:35:33.822060 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 16766
I20260926 03:35:33.839918 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:33.823023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35430 (local address 127.0.97.3:40249)
0926 03:35:33.823315 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0926 03:35:33.823325 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:35:33.823357 (+ 32us) server_negotiation.cc:407] Connection header received
0926 03:35:33.823449 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:35:33.823456 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:35:33.823560 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:35:33.823713 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:35:33.825153 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:33.826479 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:33.827817 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:35:33.828790 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:35:33.828947 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:35:33.831781 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:35:33.831798 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:35:33.831830 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:35:33.831881 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:35:33.836282 (+ 4401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:35:33.837164 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:35:33.837182 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:35:33.837188 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:35:33.837432 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:35:33.837906 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:35:33.837912 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:35:33.837915 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:35:33.838505 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:35:33.838950 (+ 445us) server_negotiation.cc:1035] Waiting for connection context
0926 03:35:33.839140 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 03:35:33.839432 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 03:35:33.840965 16889 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:35:33.986387 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 16766
I20260926 03:35:34.008462 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 16412
I20260926 03:35:34.009853 16475 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:35:34.141776 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 16412
I20260926 03:35:54.168100 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43539
--webserver_interface=127.0.97.62
--webserver_port=42683
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:35:54.563997 16939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:35:54.564383 16939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:35:54.564579 16939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:35:54.572227 16939 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 03:35:54.572359 16939 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 03:35:54.572456 16939 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 03:35:54.572556 16939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:35:54.572676 16939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:35:54.572781 16939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:35:54.572870 16939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:35:54.572950 16939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:35:54.583648 16939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=42683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:35:54.585314 16939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:35:54.586869 16939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:35:54.600314 16944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:35:54.600360 16945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:35:54.600270 16947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:35:54.602000 16939 server_base.cc:1034] running on GCE node
I20260926 03:35:54.602990 16939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:35:54.604974 16939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:35:54.606235 16939 hybrid_clock.cc:648] HybridClock initialized: now 1790393754606150 us; error 48 us; skew 500 ppm
I20260926 03:35:54.606643 16939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:35:54 dist-test-slave-dt7p krb5kdc[15859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:35:54.610241 16939 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:35:54.611768 16939 webserver.cc:460] Webserver started at http://127.0.97.62:42683/ using document root <none> and password file <none>
I20260926 03:35:54.612535 16939 fs_manager.cc:359] Metadata directory not provided
I20260926 03:35:54.612701 16939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:35:54.618849 16939 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 03:35:54.621946 16954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:35:54.623464 16939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:35:54.623617 16939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "5e2037cbf110405d90315bdb3e5d87c4"
format_stamp: "Formatted at 2026-09-26 03:35:08 on dist-test-slave-dt7p"
server_key: "dd02b3f03e7d752593de335f20be849a"
server_key_iv: "c7fb82582765c1f9b337292ae70e5c36"
server_key_version: "encryptionkey@0"
I20260926 03:35:54.624761 16939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:35:54.646477 16939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:35:54.651072 16939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:35:54.651346 16939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:35:54.672215 16939 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43539
I20260926 03:35:54.672315 17005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43539 every 8 connection(s)
I20260926 03:35:54.673992 16939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:35:54.677415 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 16939
I20260926 03:35:54.678436 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:45301
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:43539
--webserver_port=34903
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:35:54.682336 17006 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:35:54.686121 17006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4: Bootstrap starting.
I20260926 03:35:54.719229 17006 log.cc:824] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4: Log is configured to *not* fsync() on all Append() calls
I20260926 03:35:54.741465 17006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:35:54.742237 17006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4: Bootstrap complete.
I20260926 03:35:54.758741 17006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e2037cbf110405d90315bdb3e5d87c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43539 } }
I20260926 03:35:54.760171 17006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e2037cbf110405d90315bdb3e5d87c4, State: Initialized, Role: FOLLOWER
I20260926 03:35:54.761055 17006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e2037cbf110405d90315bdb3e5d87c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43539 } }
I20260926 03:35:54.761376 17006 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:35:54.761507 17006 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:35:54.761715 17006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [term 2 FOLLOWER]: Advancing to term 3
I20260926 03:35:54.766880 17006 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e2037cbf110405d90315bdb3e5d87c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43539 } }
I20260926 03:35:54.767478 17006 leader_election.cc:302] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e2037cbf110405d90315bdb3e5d87c4; no voters:
I20260926 03:35:54.768296 17006 leader_election.cc:288] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 03:35:54.768460 17010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 03:35:54.769640 17010 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [term 3 LEADER]: Becoming Leader. State: Replica: 5e2037cbf110405d90315bdb3e5d87c4, State: Running, Role: LEADER
I20260926 03:35:54.770339 17010 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e2037cbf110405d90315bdb3e5d87c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43539 } }
I20260926 03:35:54.770807 17006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:35:54.773334 17012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e2037cbf110405d90315bdb3e5d87c4. Latest consensus state: current_term: 3 leader_uuid: "5e2037cbf110405d90315bdb3e5d87c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e2037cbf110405d90315bdb3e5d87c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43539 } } }
I20260926 03:35:54.773782 17012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [sys.catalog]: This master's current role is: LEADER
I20260926 03:35:54.774381 17011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5e2037cbf110405d90315bdb3e5d87c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e2037cbf110405d90315bdb3e5d87c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43539 } } }
I20260926 03:35:54.774765 17011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4 [sys.catalog]: This master's current role is: LEADER
I20260926 03:35:54.779644 17015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:35:54.789464 17015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:35:54.791209 17015 catalog_manager.cc:1259] Loaded cluster ID: 7d88b04aedf44fffa5e35a9554ef5ec7
I20260926 03:35:54.791365 17015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:35:54.797348 17015 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:35:54.799559 17015 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4: Loaded TSK: 0
I20260926 03:35:54.799991 17015 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5e2037cbf110405d90315bdb3e5d87c4: Loaded TSK: 1
F20260926 03:35:54.801316 17015 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7efdcc2ae9ad google::LogMessage::Fail() at ??:0
@ 0x7efdcc2b188a google::LogMessage::SendToLog() at ??:0
@ 0x7efdcc2ae4de google::LogMessage::Flush() at ??:0
@ 0x7efdcc2afa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7efdd00127d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7efdd000173e kudu::security::TokenSigner::ImportKeys() at ??:0
I20260926 03:35:54.822332 17028 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7efde1cdd1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7efde1cdc783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7efde1d682a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7efde1d680c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7efde1db25e9 std::function<>::operator()() at ??:0
@ 0x7efde1ce0cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7efde1cdec36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7efde1d66610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7efde1d66439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7efdceefa345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7efdceefd91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7efdceefd799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7efdceed8eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7efdd21b26db start_thread at ??:0
@ 0x7efdcb14861f clone at ??:0
W20260926 03:35:55.152992 17008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:35:55.153430 17008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:35:55.153610 17008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:35:55.161893 17008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:35:55.162117 17008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:35:55.162318 17008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:35:55.173498 17008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:45301
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=34903
--tserver_master_addrs=127.0.97.62:43539
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:35:55.175269 17008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:35:55.176913 17008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:35:55.191779 17053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:35:55.191803 17054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:35:55.191871 17056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:35:55.197863 17008 server_base.cc:1034] running on GCE node
I20260926 03:35:55.198635 17008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:35:55.199831 17008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:35:55.201036 17008 hybrid_clock.cc:648] HybridClock initialized: now 1790393755200979 us; error 43 us; skew 500 ppm
I20260926 03:35:55.201407 17008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:35:55 dist-test-slave-dt7p krb5kdc[15859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:35:55.204909 17008 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:35:55.206650 17008 webserver.cc:460] Webserver started at http://127.0.97.1:34903/ using document root <none> and password file <none>
I20260926 03:35:55.207518 17008 fs_manager.cc:359] Metadata directory not provided
I20260926 03:35:55.207677 17008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:35:55.214213 17008 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260926 03:35:55.217450 17063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:35:55.219102 17008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:35:55.219316 17008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/wal
uuid: "274c847118904ebe8f3142653783b177"
format_stamp: "Formatted at 2026-09-26 03:35:08 on dist-test-slave-dt7p"
server_key: "a79714ed783fa6b911a7325785da2249"
server_key_iv: "7bb97d1072f73851ed64094c80cb3866"
server_key_version: "encryptionkey@0"
I20260926 03:35:55.220489 17008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:35:55.246107 17008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:35:55.250777 17008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:35:55.251106 17008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:35:55.252362 17008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:35:55.253912 17008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:35:55.254043 17008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:35:55.254184 17008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:35:55.254289 17008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:35:55.294325 17008 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45301
I20260926 03:35:55.294457 17175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45301 every 8 connection(s)
I20260926 03:35:55.296033 17008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:35:55.301574 17178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:55.299426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.1:39961)
0926 03:35:55.300266 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:55.300503 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":325,"threads_started":1}
W20260926 03:35:55.302238 17176 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43539 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:43539: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
I20260926 03:35:55.303026 17178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:55.302725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.1:40339)
0926 03:35:55.302854 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:55.302917 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 03:35:55.303771 17178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:55.303480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.1:41667)
0926 03:35:55.303592 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:55.303665 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 03:35:55.303993 17176 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 03:35:55.305176 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 17008
I20260926 03:35:55.305886 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:44503
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:43539
--webserver_port=40911
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:35:55.690670 17179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:35:55.691237 17179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:35:55.691414 17179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:35:55.699347 17179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:35:55.699474 17179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:35:55.699702 17179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:35:55.710404 17179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:44503
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=40911
--tserver_master_addrs=127.0.97.62:43539
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:35:55.712122 17179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:35:55.713773 17179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:35:55.728657 17187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:35:55.728623 17185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:35:55.728556 17184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:35:55.734122 17179 server_base.cc:1034] running on GCE node
I20260926 03:35:55.734820 17179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:35:55.735883 17179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:35:55.737088 17179 hybrid_clock.cc:648] HybridClock initialized: now 1790393755737017 us; error 60 us; skew 500 ppm
I20260926 03:35:55.737447 17179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:35:55 dist-test-slave-dt7p krb5kdc[15859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:35:55.741062 17179 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:35:55.742715 17179 webserver.cc:460] Webserver started at http://127.0.97.2:40911/ using document root <none> and password file <none>
I20260926 03:35:55.743605 17179 fs_manager.cc:359] Metadata directory not provided
I20260926 03:35:55.743760 17179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:35:55.749820 17179 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 03:35:55.753156 17194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:35:55.755460 17179 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260926 03:35:55.755653 17179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/wal
uuid: "36181f43ae0844ea85f82c508cfdec95"
format_stamp: "Formatted at 2026-09-26 03:35:09 on dist-test-slave-dt7p"
server_key: "6dbeb6dedc2d5978314e0b05541991fc"
server_key_iv: "5e13ce2905cf19692b7d5f56a62c17df"
server_key_version: "encryptionkey@0"
I20260926 03:35:55.756954 17179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:35:55.781221 17179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:35:55.785876 17179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:35:55.786147 17179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:35:55.787432 17179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:35:55.789037 17179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:35:55.789131 17179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:35:55.789229 17179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:35:55.789287 17179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:35:55.827145 17179 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44503
I20260926 03:35:55.827240 17306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44503 every 8 connection(s)
I20260926 03:35:55.828922 17179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:35:55.834821 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:55.832637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.2:40167)
0926 03:35:55.833528 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:55.833771 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":316,"threads_started":1}
I20260926 03:35:55.835492 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 17179
W20260926 03:35:55.835525 17307 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43539 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:43539: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
I20260926 03:35:55.836313 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:40249
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:43539
--webserver_port=44143
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:35:55.836467 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:55.836035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.2:50419)
0926 03:35:55.836225 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:55.836319 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 03:35:55.837492 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:55.837010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.2:43113)
0926 03:35:55.837194 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:55.837322 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 03:35:55.837757 17307 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 03:35:56.215508 17310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:35:56.215862 17310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:35:56.216039 17310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:35:56.224187 17310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:35:56.224351 17310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:35:56.224548 17310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:35:56.235611 17310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:40249
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=44143
--tserver_master_addrs=127.0.97.62:43539
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:35:56.237188 17310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:35:56.238816 17310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:35:56.253044 17315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:35:56.253094 17316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:35:56.253151 17318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:35:56.254218 17310 server_base.cc:1034] running on GCE node
I20260926 03:35:56.254817 17310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:35:56.255790 17310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:35:56.257015 17310 hybrid_clock.cc:648] HybridClock initialized: now 1790393756256968 us; error 29 us; skew 500 ppm
I20260926 03:35:56.257328 17310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:35:56 dist-test-slave-dt7p krb5kdc[15859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:35:56.260658 17310 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:35:56.262240 17310 webserver.cc:460] Webserver started at http://127.0.97.3:44143/ using document root <none> and password file <none>
I20260926 03:35:56.263005 17310 fs_manager.cc:359] Metadata directory not provided
I20260926 03:35:56.263113 17310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:35:56.269048 17310 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 03:35:56.271940 17325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:35:56.273561 17310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:35:56.273746 17310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/wal
uuid: "157e59415ba045a5815168d80ffa59b1"
format_stamp: "Formatted at 2026-09-26 03:35:10 on dist-test-slave-dt7p"
server_key: "52c4d483c830ebdffc58ec3f2df12a18"
server_key_iv: "064ec938c8c2670e097170ab00301d57"
server_key_version: "encryptionkey@0"
I20260926 03:35:56.274809 17310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:35:56.289875 17310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:35:56.294353 17310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:35:56.294663 17310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:35:56.295758 17310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:35:56.297379 17310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:35:56.297503 17310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:35:56.297641 17310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:35:56.297747 17310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:35:56.305858 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:56.304786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.1:53581)
0926 03:35:56.305403 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:56.305503 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":329,"threads_started":1}
I20260926 03:35:56.335860 17310 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40249
I20260926 03:35:56.335918 17438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40249 every 8 connection(s)
I20260926 03:35:56.337702 17310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:35:56.343173 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:56.341064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.3:45099)
0926 03:35:56.341903 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:56.342127 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":313,"threads_started":1}
W20260926 03:35:56.343954 17439 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43539 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:43539: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
I20260926 03:35:56.344965 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:56.344520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.3:43763)
0926 03:35:56.344748 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:56.344840 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260926 03:35:56.345903 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:56.345493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.3:45147)
0926 03:35:56.345650 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:56.345753 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 03:35:56.345968 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 17310
W20260926 03:35:56.346187 17439 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 03:35:56.839295 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:56.838400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.2:43657)
0926 03:35:56.838932 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:56.839029 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":262,"threads_started":1}
I20260926 03:35:57.307873 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:57.306867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.1:51477)
0926 03:35:57.307545 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:57.307639 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":378,"threads_started":1}
I20260926 03:35:57.348189 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:57.346853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.3:59161)
0926 03:35:57.347513 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:57.347736 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":388,"threads_started":1}
I20260926 03:35:57.841358 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:57.840456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.2:54705)
0926 03:35:57.841019 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:57.841145 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":280,"threads_started":1}
I20260926 03:35:58.309813 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:58.308947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.1:42451)
0926 03:35:58.309496 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:58.309584 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":292,"threads_started":1}
I20260926 03:35:58.350311 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:58.349314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.3:49787)
0926 03:35:58.349942 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:58.350050 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":349,"threads_started":1}
I20260926 03:35:58.843392 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:58.842536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.2:35811)
0926 03:35:58.843053 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:58.843155 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":265,"threads_started":1}
I20260926 03:35:59.311945 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:59.311066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.1:50849)
0926 03:35:59.311625 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:59.311720 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":285,"threads_started":1}
I20260926 03:35:59.352084 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:59.351314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.3:43917)
0926 03:35:59.351784 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:59.351874 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":250,"threads_started":1}
I20260926 03:35:59.845615 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:35:59.844666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.2:47173)
0926 03:35:59.845289 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0926 03:35:59.845390 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":329,"threads_started":1}
I20260926 03:36:00.314417 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:00.313364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.1:44473)
0926 03:36:00.314082 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:00.314192 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":386,"threads_started":1}
I20260926 03:36:00.354223 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:00.353300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.3:59891)
0926 03:36:00.353856 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:00.353968 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":274,"threads_started":1}
I20260926 03:36:00.847645 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:00.846742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.2:40113)
0926 03:36:00.847317 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:00.847422 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":299,"threads_started":1}
I20260926 03:36:01.316514 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:01.315552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.1:46057)
0926 03:36:01.316132 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:01.316275 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":305,"threads_started":1}
I20260926 03:36:01.356257 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:01.355346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.3:38309)
0926 03:36:01.355946 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:01.356041 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":308,"threads_started":1}
I20260926 03:36:01.849681 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:01.848764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.2:50515)
0926 03:36:01.849316 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:01.849449 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":259,"threads_started":1}
I20260926 03:36:02.318738 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:02.317731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.1:45343)
0926 03:36:02.318347 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:02.318450 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":257,"threads_started":1}
I20260926 03:36:02.358239 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:02.357405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.3:57889)
0926 03:36:02.357927 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:02.358033 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":260,"threads_started":1}
I20260926 03:36:02.851759 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:02.850858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.2:48307)
0926 03:36:02.851409 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:02.851500 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":233,"threads_started":1}
I20260926 03:36:03.320859 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:03.319959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.1:41145)
0926 03:36:03.320510 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:03.320631 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":297,"threads_started":1}
I20260926 03:36:03.360482 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:03.359441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.3:51195)
0926 03:36:03.360074 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:03.360206 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":294,"threads_started":1}
I20260926 03:36:03.853756 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:03.852908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.2:53161)
0926 03:36:03.853460 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:03.853551 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":297,"threads_started":1}
I20260926 03:36:04.322926 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:04.321937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.1:44913)
0926 03:36:04.322518 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:04.322646 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":298,"threads_started":1}
I20260926 03:36:04.362455 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:04.361667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.3:47239)
0926 03:36:04.362169 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:04.362256 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":245,"threads_started":1}
I20260926 03:36:04.855706 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:04.854812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.2:48175)
0926 03:36:04.855389 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:04.855486 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":263,"threads_started":1}
I20260926 03:36:05.325130 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:05.324139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.1:34241)
0926 03:36:05.324758 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:05.324870 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":316,"threads_started":1}
I20260926 03:36:05.364550 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:05.363501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.3:44513)
0926 03:36:05.364183 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:05.364317 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":382,"threads_started":1}
I20260926 03:36:05.857874 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:05.856854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.2:53071)
0926 03:36:05.857531 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:05.857644 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":394,"threads_started":1}
I20260926 03:36:06.327128 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:06.326246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.1:52729)
0926 03:36:06.326797 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:06.326903 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":277,"threads_started":1}
I20260926 03:36:06.366704 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:06.365825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.3:57451)
0926 03:36:06.366381 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:06.366486 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":322,"threads_started":1}
I20260926 03:36:06.859807 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:06.858971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.2:51097)
0926 03:36:06.859464 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:06.859567 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":247,"threads_started":1}
I20260926 03:36:07.329411 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:07.328396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.1:59997)
0926 03:36:07.329075 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:07.329200 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":368,"threads_started":1}
I20260926 03:36:07.368894 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:07.367946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.3:33005)
0926 03:36:07.368480 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:07.368670 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":255,"threads_started":1}
I20260926 03:36:07.861930 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:07.861076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.2:48471)
0926 03:36:07.861605 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:07.861719 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":262,"threads_started":1}
I20260926 03:36:08.331540 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:08.330485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.1:36969)
0926 03:36:08.331168 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:08.331284 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":316,"threads_started":1}
I20260926 03:36:08.370743 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:08.369917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.3:43507)
0926 03:36:08.370444 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:08.370543 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":278,"threads_started":1}
I20260926 03:36:08.864144 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:08.863152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.2:51003)
0926 03:36:08.863803 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:08.863898 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":367,"threads_started":1}
I20260926 03:36:09.333822 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:09.332754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.1:49203)
0926 03:36:09.333409 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:09.333517 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":349,"threads_started":1}
I20260926 03:36:09.372911 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:09.371880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.3:57925)
0926 03:36:09.372483 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:09.372662 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":292,"threads_started":1}
I20260926 03:36:09.866016 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:09.865164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.2:57511)
0926 03:36:09.865678 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:09.865771 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":270,"threads_started":1}
I20260926 03:36:10.336143 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:10.335257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.1:56629)
0926 03:36:10.335816 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:10.335912 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":287,"threads_started":1}
I20260926 03:36:10.374838 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:10.373964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.3:43131)
0926 03:36:10.374496 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:10.374599 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":275,"threads_started":1}
I20260926 03:36:10.868011 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:10.867222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.2:38347)
0926 03:36:10.867695 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:10.867781 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":248,"threads_started":1}
I20260926 03:36:11.338297 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:11.337307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.1:45821)
0926 03:36:11.337999 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:11.338105 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":343,"threads_started":1}
I20260926 03:36:11.354853 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 17008
I20260926 03:36:11.377331 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:11.376213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43539 (local address 127.0.97.3:58937)
0926 03:36:11.376884 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:11.377011 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":345,"threads_started":1}
I20260926 03:36:11.382865 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:11.356693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35610 (local address 127.0.97.1:45301)
0926 03:36:11.357498 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:11.357517 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:11.357572 (+ 55us) server_negotiation.cc:407] Connection header received
0926 03:36:11.359497 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:11.359509 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:11.360088 (+ 579us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:11.360646 (+ 558us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:11.362432 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:11.364121 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:11.365567 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:11.366435 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:11.366628 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:11.374465 (+ 7837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:11.374478 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:11.374503 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:36:11.374562 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:11.378966 (+ 4404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:11.380005 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:11.380014 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:11.380023 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:11.380217 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:11.380652 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:11.380656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:11.380658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:11.381353 (+ 695us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:11.381618 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:11.382004 (+ 386us) server_negotiation.cc:299] Negotiation successful
0926 03:36:11.382529 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":387,"threads_started":1}
I20260926 03:36:11.384552 17171 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:36:11.476173 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 17008
I20260926 03:36:11.496907 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 17179
I20260926 03:36:11.516412 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:11.498004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52502 (local address 127.0.97.2:44503)
0926 03:36:11.498657 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:11.498676 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:11.498723 (+ 47us) server_negotiation.cc:407] Connection header received
0926 03:36:11.500481 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:11.500492 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:11.501097 (+ 605us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:11.501504 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:11.503112 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:11.504484 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:11.505957 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:11.506704 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:11.506920 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:11.509777 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:11.509785 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:11.509805 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:36:11.509845 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:11.512967 (+ 3122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:11.513667 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:11.513674 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:11.513680 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:11.513853 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:11.514228 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:11.514233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:11.514237 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:11.514799 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:11.515132 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:11.515614 (+ 482us) server_negotiation.cc:299] Negotiation successful
0926 03:36:11.516075 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":322,"threads_started":1}
I20260926 03:36:11.518054 17302 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:36:11.609359 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 17179
I20260926 03:36:11.629671 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 17310
I20260926 03:36:11.649937 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:11.630984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57636 (local address 127.0.97.3:40249)
0926 03:36:11.632250 (+ 1266us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:11.632270 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:11.632320 (+ 50us) server_negotiation.cc:407] Connection header received
0926 03:36:11.634287 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:11.634299 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:11.634826 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:11.635158 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:11.636670 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:11.638043 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:11.639426 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:11.640159 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:11.640298 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:11.643294 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:11.643303 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:11.643324 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:36:11.643372 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:11.646621 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:11.647255 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:11.647263 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:11.647270 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:11.647441 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:11.647805 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:11.647809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:11.647810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:11.648407 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:11.648744 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:11.649110 (+ 366us) server_negotiation.cc:299] Negotiation successful
0926 03:36:11.649586 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1087,"thread_start_us":912,"threads_started":1}
I20260926 03:36:11.651568 17434 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:36:11.742689 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 17310
2026-09-26T03:36:11Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65456 ms)
[----------] 30 tests from SecurityITest (308479 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:36:11 dist-test-slave-dt7p krb5kdc[17496](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:36:11 dist-test-slave-dt7p krb5kdc[17496](info): set up 2 sockets
Sep 26 03:36:11 dist-test-slave-dt7p krb5kdc[17496](info): commencing operation
krb5kdc: starting...
W20260926 03:36:12.913059 388 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:36:13 dist-test-slave-dt7p krb5kdc[17496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:36:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:36:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:36:13.202481 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35481
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35481
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:36:13.590252 17509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:36:13.590600 17509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:36:13.590699 17509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:36:13.598508 17509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:36:13.598652 17509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:36:13.598703 17509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:36:13.598768 17509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:36:13.598816 17509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:36:13.609787 17509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:36:13.611459 17509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:36:13.613051 17509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:36:13.626852 17515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:13.626922 17514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:13.627614 17517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:36:13.627813 17509 server_base.cc:1034] running on GCE node
I20260926 03:36:13.628894 17509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:36:13.630599 17509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:36:13.631819 17509 hybrid_clock.cc:648] HybridClock initialized: now 1790393773631768 us; error 35 us; skew 500 ppm
I20260926 03:36:13.632246 17509 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:36:13 dist-test-slave-dt7p krb5kdc[17496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:36:13.635783 17509 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:36:13.637552 17509 webserver.cc:460] Webserver started at http://127.0.97.62:37667/ using document root <none> and password file <none>
I20260926 03:36:13.638319 17509 fs_manager.cc:359] Metadata directory not provided
I20260926 03:36:13.638420 17509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:36:13.638734 17509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:36:13.642197 17509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "6700b06e9dd44bfca8f46f690649f3ef"
format_stamp: "Formatted at 2026-09-26 03:36:13 on dist-test-slave-dt7p"
server_key: "8bd07f2443ff8ea32881b9a4894dc8b8"
server_key_iv: "529232568001566d8b7557e1721a89c9"
server_key_version: "encryptionkey@0"
I20260926 03:36:13.643000 17509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "6700b06e9dd44bfca8f46f690649f3ef"
format_stamp: "Formatted at 2026-09-26 03:36:13 on dist-test-slave-dt7p"
server_key: "8bd07f2443ff8ea32881b9a4894dc8b8"
server_key_iv: "529232568001566d8b7557e1721a89c9"
server_key_version: "encryptionkey@0"
I20260926 03:36:13.648945 17509 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260926 03:36:13.652801 17524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:13.654342 17509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:36:13.654538 17509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "6700b06e9dd44bfca8f46f690649f3ef"
format_stamp: "Formatted at 2026-09-26 03:36:13 on dist-test-slave-dt7p"
server_key: "8bd07f2443ff8ea32881b9a4894dc8b8"
server_key_iv: "529232568001566d8b7557e1721a89c9"
server_key_version: "encryptionkey@0"
I20260926 03:36:13.654767 17509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:36:13.675658 17509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:36:13.680188 17509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:36:13.680595 17509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:36:13.701689 17509 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35481
I20260926 03:36:13.701720 17575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35481 every 8 connection(s)
I20260926 03:36:13.703399 17509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:36:13.708277 17576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:36:13.712071 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 17509
I20260926 03:36:13.712339 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:36:13.713166 388 external_mini_cluster.cc:1428] Setting key a1fa550e69d5a48902ab938ea367e292
I20260926 03:36:13.723433 17576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6700b06e9dd44bfca8f46f690649f3ef: Bootstrap starting.
I20260926 03:36:13.727607 17576 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6700b06e9dd44bfca8f46f690649f3ef: Neither blocks nor log segments found. Creating new log.
I20260926 03:36:13.728859 17576 log.cc:824] T 00000000000000000000000000000000 P 6700b06e9dd44bfca8f46f690649f3ef: Log is configured to *not* fsync() on all Append() calls
Sep 26 03:36:13 dist-test-slave-dt7p krb5kdc[17496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:36:13.732415 17576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6700b06e9dd44bfca8f46f690649f3ef: No bootstrap required, opened a new log
I20260926 03:36:13.739264 17579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:13.715617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55540 (local address 127.0.97.62:35481)
0926 03:36:13.716538 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:13.716552 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:13.716651 (+ 99us) server_negotiation.cc:407] Connection header received
0926 03:36:13.718619 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:13.718658 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:13.719311 (+ 653us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:13.719829 (+ 518us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:13.721390 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:13.723483 (+ 2093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:13.725074 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:13.726228 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:13.726415 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:13.731197 (+ 4782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:13.731205 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:13.731221 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:36:13.731259 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:13.734738 (+ 3479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:13.735618 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:13.735624 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:13.735634 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:13.735802 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:13.736226 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:13.736229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:13.736231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:13.736905 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:13.737139 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:13.737620 (+ 481us) server_negotiation.cc:299] Negotiation successful
0926 03:36:13.738034 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":414,"threads_started":1}
I20260926 03:36:13.742520 17576 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6700b06e9dd44bfca8f46f690649f3ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6700b06e9dd44bfca8f46f690649f3ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35481 } }
I20260926 03:36:13.742949 17576 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6700b06e9dd44bfca8f46f690649f3ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:36:13.743059 17576 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6700b06e9dd44bfca8f46f690649f3ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6700b06e9dd44bfca8f46f690649f3ef, State: Initialized, Role: FOLLOWER
I20260926 03:36:13.743711 17576 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6700b06e9dd44bfca8f46f690649f3ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6700b06e9dd44bfca8f46f690649f3ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35481 } }
I20260926 03:36:13.743988 17576 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6700b06e9dd44bfca8f46f690649f3ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:36:13.744143 17576 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6700b06e9dd44bfca8f46f690649f3ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:36:13.744318 17576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6700b06e9dd44bfca8f46f690649f3ef [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:36:13.746788 17576 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6700b06e9dd44bfca8f46f690649f3ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6700b06e9dd44bfca8f46f690649f3ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35481 } }
I20260926 03:36:13.747349 17576 leader_election.cc:302] T 00000000000000000000000000000000 P 6700b06e9dd44bfca8f46f690649f3ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6700b06e9dd44bfca8f46f690649f3ef; no voters:
I20260926 03:36:13.747988 17576 leader_election.cc:288] T 00000000000000000000000000000000 P 6700b06e9dd44bfca8f46f690649f3ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:36:13.748170 17581 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6700b06e9dd44bfca8f46f690649f3ef [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:36:13.749749 17581 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6700b06e9dd44bfca8f46f690649f3ef [term 1 LEADER]: Becoming Leader. State: Replica: 6700b06e9dd44bfca8f46f690649f3ef, State: Running, Role: LEADER
I20260926 03:36:13.750309 17581 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6700b06e9dd44bfca8f46f690649f3ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6700b06e9dd44bfca8f46f690649f3ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35481 } }
I20260926 03:36:13.750614 17576 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6700b06e9dd44bfca8f46f690649f3ef [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:36:13.753597 17583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6700b06e9dd44bfca8f46f690649f3ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6700b06e9dd44bfca8f46f690649f3ef. Latest consensus state: current_term: 1 leader_uuid: "6700b06e9dd44bfca8f46f690649f3ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6700b06e9dd44bfca8f46f690649f3ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35481 } } }
I20260926 03:36:13.754123 17583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6700b06e9dd44bfca8f46f690649f3ef [sys.catalog]: This master's current role is: LEADER
I20260926 03:36:13.753646 17582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6700b06e9dd44bfca8f46f690649f3ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6700b06e9dd44bfca8f46f690649f3ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6700b06e9dd44bfca8f46f690649f3ef" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35481 } } }
I20260926 03:36:13.754420 17582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6700b06e9dd44bfca8f46f690649f3ef [sys.catalog]: This master's current role is: LEADER
I20260926 03:36:13.755904 17590 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:36:13.763181 17590 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:36:13.778964 17590 catalog_manager.cc:1347] Generated new cluster ID: 83eb73cac363441d8ca5b49d32de5384
I20260926 03:36:13.779122 17590 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:36:13.809522 17590 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:36:13.811180 17590 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:36:13.822782 17590 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6700b06e9dd44bfca8f46f690649f3ef: Generated new TSK 0
I20260926 03:36:13.823876 17590 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:36:13.905354 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35481
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:36:14.297386 17602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:36:14.297788 17602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:36:14.297978 17602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:36:14.305948 17602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:36:14.306165 17602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:36:14.306371 17602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:36:14.317229 17602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:36:14.318948 17602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:36:14.320690 17602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:36:14.337105 17607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:14.337006 17608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:14.337368 17610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:36:14.338109 17602 server_base.cc:1034] running on GCE node
I20260926 03:36:14.338784 17602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:36:14.339895 17602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:36:14.341095 17602 hybrid_clock.cc:648] HybridClock initialized: now 1790393774341022 us; error 62 us; skew 500 ppm
I20260926 03:36:14.341456 17602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:36:14 dist-test-slave-dt7p krb5kdc[17496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:36:14.345098 17602 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:36:14.347004 17602 webserver.cc:460] Webserver started at http://127.0.97.1:35453/ using document root <none> and password file <none>
I20260926 03:36:14.347831 17602 fs_manager.cc:359] Metadata directory not provided
I20260926 03:36:14.347939 17602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:36:14.348244 17602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:36:14.351774 17602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-0/data/instance:
uuid: "17e665bbeeb94346820b3ea46678d64b"
format_stamp: "Formatted at 2026-09-26 03:36:14 on dist-test-slave-dt7p"
server_key: "06ba1aed20fc5e740b4b02db79bf0616"
server_key_iv: "4ac959aae7dd63c2b44cbb619c8d71a0"
server_key_version: "encryptionkey@0"
I20260926 03:36:14.352752 17602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-0/wal/instance:
uuid: "17e665bbeeb94346820b3ea46678d64b"
format_stamp: "Formatted at 2026-09-26 03:36:14 on dist-test-slave-dt7p"
server_key: "06ba1aed20fc5e740b4b02db79bf0616"
server_key_iv: "4ac959aae7dd63c2b44cbb619c8d71a0"
server_key_version: "encryptionkey@0"
I20260926 03:36:14.359103 17602 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260926 03:36:14.363564 17617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:14.365303 17602 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260926 03:36:14.365513 17602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-0/wal
uuid: "17e665bbeeb94346820b3ea46678d64b"
format_stamp: "Formatted at 2026-09-26 03:36:14 on dist-test-slave-dt7p"
server_key: "06ba1aed20fc5e740b4b02db79bf0616"
server_key_iv: "4ac959aae7dd63c2b44cbb619c8d71a0"
server_key_version: "encryptionkey@0"
I20260926 03:36:14.365783 17602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:36:14.394500 17602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:36:14.399995 17602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:36:14.400370 17602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:36:14.401655 17602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:36:14.403214 17602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:36:14.403292 17602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:14.403391 17602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:36:14.403438 17602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:14.440670 17602 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45523
I20260926 03:36:14.440706 17729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45523 every 8 connection(s)
I20260926 03:36:14.442263 17602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:36:14.448189 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 17602
I20260926 03:36:14.448427 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:36:14.449347 388 external_mini_cluster.cc:1428] Setting key 2c9030c70ad6745e216128f153952c3c
Sep 26 03:36:14 dist-test-slave-dt7p krb5kdc[17496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:36:14.467916 17732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:14.445323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37797 (local address 127.0.97.62:35481)
0926 03:36:14.445869 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:14.445874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:14.446937 (+ 1063us) server_negotiation.cc:407] Connection header received
0926 03:36:14.449865 (+ 2928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:14.449871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:14.449960 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:14.450064 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:14.452291 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:14.453147 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:14.455873 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:14.456304 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:14.456408 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:14.461548 (+ 5140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:14.461553 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:14.461556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:36:14.461591 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:14.464454 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:14.465265 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:14.465270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:14.465281 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:14.465486 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:14.465958 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:14.465963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:14.465966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:14.466272 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:14.466441 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:14.467224 (+ 783us) server_negotiation.cc:299] Negotiation successful
0926 03:36:14.467493 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":265,"threads_started":1}
I20260926 03:36:14.469201 17733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:14.445977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35481 (local address 127.0.97.1:37797)
0926 03:36:14.446757 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:14.446801 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:14.449474 (+ 2673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:14.450375 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:14.450390 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:14.451195 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:36:14.452059 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:14.452086 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:14.453319 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:14.453328 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:14.455721 (+ 2393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:14.455741 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:14.456556 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:14.456591 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:14.456953 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:14.457823 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:36:14.457863 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:36:14.461315 (+ 3452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:36:14.464615 (+ 3300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:36:14.464626 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:36:14.464645 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:36:14.465126 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:36:14.465606 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:36:14.465613 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:36:14.465617 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:36:14.465838 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:36:14.466410 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:36:14.466419 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:36:14.466809 (+ 390us) client_negotiation.cc:769] Sending connection context
0926 03:36:14.467110 (+ 301us) client_negotiation.cc:240] Negotiation successful
0926 03:36:14.467721 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":305,"threads_started":1}
I20260926 03:36:14.471102 17730 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35481
I20260926 03:36:14.471483 17730 heartbeater.cc:464] Registering TS with master...
I20260926 03:36:14.472491 17730 heartbeater.cc:511] Master 127.0.97.62:35481 requested a full tablet report, sending...
I20260926 03:36:14.475313 17541 ts_manager.cc:194] Registered new tserver with Master: 17e665bbeeb94346820b3ea46678d64b (127.0.97.1:45523)
I20260926 03:36:14.477783 17541 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:37797
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:36:14.515772 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35481
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:36:14.898197 17736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:36:14.898551 17736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:36:14.898720 17736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:36:14.906368 17736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:36:14.906481 17736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:36:14.906674 17736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:36:14.916982 17736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:36:14.918618 17736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:36:14.920204 17736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:36:14.935319 17741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:14.935261 17742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:14.935222 17744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:36:14.936218 17736 server_base.cc:1034] running on GCE node
I20260926 03:36:14.936983 17736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:36:14.937984 17736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:36:14.939216 17736 hybrid_clock.cc:648] HybridClock initialized: now 1790393774939169 us; error 31 us; skew 500 ppm
I20260926 03:36:14.939584 17736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:36:14 dist-test-slave-dt7p krb5kdc[17496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:36:14.943871 17736 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:36:14.945839 17736 webserver.cc:460] Webserver started at http://127.0.97.2:41647/ using document root <none> and password file <none>
I20260926 03:36:14.946709 17736 fs_manager.cc:359] Metadata directory not provided
I20260926 03:36:14.946864 17736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:36:14.947238 17736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:36:14.950853 17736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-1/data/instance:
uuid: "a7a8da2dc172444882f70f43168fed87"
format_stamp: "Formatted at 2026-09-26 03:36:14 on dist-test-slave-dt7p"
server_key: "f2311a2d86cb9814e9066e04946efe9b"
server_key_iv: "e6c36773df1ac68106c0d9491b4e0220"
server_key_version: "encryptionkey@0"
I20260926 03:36:14.951840 17736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a7a8da2dc172444882f70f43168fed87"
format_stamp: "Formatted at 2026-09-26 03:36:14 on dist-test-slave-dt7p"
server_key: "f2311a2d86cb9814e9066e04946efe9b"
server_key_iv: "e6c36773df1ac68106c0d9491b4e0220"
server_key_version: "encryptionkey@0"
I20260926 03:36:14.958293 17736 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260926 03:36:14.962487 17751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:14.964250 17736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:36:14.964438 17736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-1/wal
uuid: "a7a8da2dc172444882f70f43168fed87"
format_stamp: "Formatted at 2026-09-26 03:36:14 on dist-test-slave-dt7p"
server_key: "f2311a2d86cb9814e9066e04946efe9b"
server_key_iv: "e6c36773df1ac68106c0d9491b4e0220"
server_key_version: "encryptionkey@0"
I20260926 03:36:14.964710 17736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:36:14.996340 17736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:36:15.001402 17736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:36:15.001751 17736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:36:15.002902 17736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:36:15.004547 17736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:36:15.004676 17736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 03:36:15.004849 17736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:36:15.004937 17736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:15.041139 17736 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34907
I20260926 03:36:15.041167 17863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34907 every 8 connection(s)
I20260926 03:36:15.042747 17736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:36:15.045800 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 17736
I20260926 03:36:15.046038 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:36:15.046741 388 external_mini_cluster.cc:1428] Setting key d81b3007ace1b23ec32c442ebe44d4b1
Sep 26 03:36:15 dist-test-slave-dt7p krb5kdc[17496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:36:15.067337 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:15.046247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41195 (local address 127.0.97.62:35481)
0926 03:36:15.046789 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:15.046794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:15.048184 (+ 1390us) server_negotiation.cc:407] Connection header received
0926 03:36:15.050451 (+ 2267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:15.050457 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:15.050562 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:15.050689 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:15.052553 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:15.053388 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:15.055176 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:15.055720 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:15.055877 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:15.060440 (+ 4563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:15.060445 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:15.060449 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:36:15.060488 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:15.063543 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:15.064614 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:15.064623 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:15.064640 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:15.064782 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:15.065326 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:15.065329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:15.065331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:15.065667 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:15.065786 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:15.066815 (+ 1029us) server_negotiation.cc:299] Negotiation successful
0926 03:36:15.067021 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":281,"threads_started":1}
I20260926 03:36:15.069226 17867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:15.046990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35481 (local address 127.0.97.2:41195)
0926 03:36:15.048018 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:15.048065 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:15.050146 (+ 2081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:15.050970 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:15.050979 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:15.051639 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:36:15.052367 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:15.052386 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:15.053537 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:15.053550 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:15.055031 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:15.055044 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:15.055973 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:15.055981 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:15.056233 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:15.057052 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:36:15.057076 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:36:15.060153 (+ 3077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:36:15.063709 (+ 3556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:36:15.063723 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:36:15.063747 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:36:15.064430 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:36:15.064916 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:36:15.064924 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:36:15.064930 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:36:15.065207 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:36:15.065819 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:36:15.065830 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:36:15.066376 (+ 546us) client_negotiation.cc:769] Sending connection context
0926 03:36:15.066717 (+ 341us) client_negotiation.cc:240] Negotiation successful
0926 03:36:15.067398 (+ 681us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"thread_start_us":407,"threads_started":1}
I20260926 03:36:15.071210 17864 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35481
I20260926 03:36:15.071599 17864 heartbeater.cc:464] Registering TS with master...
I20260926 03:36:15.072679 17864 heartbeater.cc:511] Master 127.0.97.62:35481 requested a full tablet report, sending...
I20260926 03:36:15.074851 17541 ts_manager.cc:194] Registered new tserver with Master: a7a8da2dc172444882f70f43168fed87 (127.0.97.2:34907)
I20260926 03:36:15.076373 17541 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41195
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:36:15.121079 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35481
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:36:15.482209 17730 heartbeater.cc:503] Master 127.0.97.62:35481 was elected leader, sending a full tablet report...
W20260926 03:36:15.509899 17870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:36:15.510219 17870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:36:15.510330 17870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:36:15.517897 17870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:36:15.518011 17870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:36:15.518136 17870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:36:15.528975 17870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:36:15.530555 17870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:36:15.532099 17870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:36:15.547230 17878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:15.547323 17876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:15.547399 17875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:36:15.548595 17870 server_base.cc:1034] running on GCE node
I20260926 03:36:15.549228 17870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:36:15.550318 17870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:36:15.551554 17870 hybrid_clock.cc:648] HybridClock initialized: now 1790393775551503 us; error 46 us; skew 500 ppm
I20260926 03:36:15.551916 17870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:36:15 dist-test-slave-dt7p krb5kdc[17496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:36:15.555509 17870 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:36:15.557137 17870 webserver.cc:460] Webserver started at http://127.0.97.3:38443/ using document root <none> and password file <none>
I20260926 03:36:15.557899 17870 fs_manager.cc:359] Metadata directory not provided
I20260926 03:36:15.558007 17870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:36:15.558369 17870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:36:15.561757 17870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-2/data/instance:
uuid: "c583e21944694ed480911d934709e269"
format_stamp: "Formatted at 2026-09-26 03:36:15 on dist-test-slave-dt7p"
server_key: "b385aec82b943403c9134d1dff67e5e4"
server_key_iv: "f5822258ab734099dc15342d4522db96"
server_key_version: "encryptionkey@0"
I20260926 03:36:15.562625 17870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c583e21944694ed480911d934709e269"
format_stamp: "Formatted at 2026-09-26 03:36:15 on dist-test-slave-dt7p"
server_key: "b385aec82b943403c9134d1dff67e5e4"
server_key_iv: "f5822258ab734099dc15342d4522db96"
server_key_version: "encryptionkey@0"
I20260926 03:36:15.568703 17870 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260926 03:36:15.572870 17885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:15.574409 17870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:36:15.574599 17870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-2/wal
uuid: "c583e21944694ed480911d934709e269"
format_stamp: "Formatted at 2026-09-26 03:36:15 on dist-test-slave-dt7p"
server_key: "b385aec82b943403c9134d1dff67e5e4"
server_key_iv: "f5822258ab734099dc15342d4522db96"
server_key_version: "encryptionkey@0"
I20260926 03:36:15.574844 17870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:36:15.591054 17870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:36:15.596052 17870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:36:15.596424 17870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:36:15.597649 17870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:36:15.599386 17870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:36:15.599493 17870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:15.599648 17870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:36:15.599728 17870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:15.638309 17870 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37973
I20260926 03:36:15.638373 17997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37973 every 8 connection(s)
I20260926 03:36:15.639992 17870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:36:15.650234 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 17870
I20260926 03:36:15.650467 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:36:15.650982 388 external_mini_cluster.cc:1428] Setting key 99af84e201be1e29e3396737d54dcfce
Sep 26 03:36:15 dist-test-slave-dt7p krb5kdc[17496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:36:15.663686 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:15.643077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45991 (local address 127.0.97.62:35481)
0926 03:36:15.643716 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:15.643721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:15.644696 (+ 975us) server_negotiation.cc:407] Connection header received
0926 03:36:15.646768 (+ 2072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:15.646774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:15.646898 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:15.647032 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:15.648811 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:15.649689 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:15.651766 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:15.652411 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:15.652607 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:15.657073 (+ 4466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:15.657079 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:15.657083 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:36:15.657128 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:15.660614 (+ 3486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:15.661350 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:15.661355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:15.661358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:15.661492 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:15.661963 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:15.661967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:15.661969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:15.662302 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:15.662429 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:15.663179 (+ 750us) server_negotiation.cc:299] Negotiation successful
0926 03:36:15.663375 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":334,"threads_started":1}
I20260926 03:36:15.665001 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:15.643608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35481 (local address 127.0.97.3:45991)
0926 03:36:15.644458 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:15.644500 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:15.646455 (+ 1955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:15.647309 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:15.647318 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:15.647903 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:36:15.648555 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:15.648639 (+ 84us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:15.649852 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:15.649859 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:15.651608 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:15.651624 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:15.652719 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:15.652730 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:15.652987 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:15.653909 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:36:15.653936 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:36:15.656826 (+ 2890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:36:15.660752 (+ 3926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:36:15.660760 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:36:15.660776 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:36:15.661217 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:36:15.661606 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:36:15.661610 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:36:15.661612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:36:15.661820 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:36:15.662414 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:36:15.662421 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:36:15.662794 (+ 373us) client_negotiation.cc:769] Sending connection context
0926 03:36:15.663045 (+ 251us) client_negotiation.cc:240] Negotiation successful
0926 03:36:15.663506 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":326,"threads_started":1}
I20260926 03:36:15.666990 17998 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35481
I20260926 03:36:15.667425 17998 heartbeater.cc:464] Registering TS with master...
I20260926 03:36:15.668347 17998 heartbeater.cc:511] Master 127.0.97.62:35481 requested a full tablet report, sending...
I20260926 03:36:15.670295 17541 ts_manager.cc:194] Registered new tserver with Master: c583e21944694ed480911d934709e269 (127.0.97.3:37973)
I20260926 03:36:15.671298 17541 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45991
I20260926 03:36:15.679829 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:36:15.699949 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:15.685318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55544 (local address 127.0.97.62:35481)
0926 03:36:15.685597 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:15.685605 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:15.685951 (+ 346us) server_negotiation.cc:407] Connection header received
0926 03:36:15.686103 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:15.686107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:15.686186 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:15.686299 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:15.687859 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:15.688757 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:15.690239 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:15.690706 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:15.690849 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:15.693835 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:15.693841 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:15.693845 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:36:15.693883 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:15.696805 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:15.697554 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:15.697562 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:15.697566 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:15.697719 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:15.698153 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:15.698160 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:15.698163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:15.698634 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:15.698942 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:15.699283 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 03:36:15.699567 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 03:36:15.705204 17541 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:36:15.713336 17541 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:36:15.741154 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:15.727215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34907 (local address 127.0.0.1:60854)
0926 03:36:15.729200 (+ 1985us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:15.729224 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:15.729371 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:15.730783 (+ 1412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:15.730792 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:15.730829 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:36:15.731358 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:15.731377 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:15.734359 (+ 2982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:15.734366 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:15.737642 (+ 3276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:15.737665 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:15.740045 (+ 2380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:15.740052 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:15.740275 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:15.740356 (+ 81us) client_negotiation.cc:769] Sending connection context
0926 03:36:15.740473 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 03:36:15.740712 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1814,"thread_start_us":1619,"threads_started":1}
I20260926 03:36:15.742498 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:15.727684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60854 (local address 127.0.97.2:34907)
0926 03:36:15.730007 (+ 2323us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:15.730018 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:15.730051 (+ 33us) server_negotiation.cc:407] Connection header received
0926 03:36:15.730191 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:15.730202 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:15.730526 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:15.731581 (+ 1055us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:36:15.731918 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:15.734177 (+ 2259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:15.737964 (+ 3787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:15.739788 (+ 1824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:15.741582 (+ 1794us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:15.741857 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:15.741943 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 03:36:15.742093 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2166,"thread_start_us":2029,"threads_started":1}
I20260926 03:36:15.747486 17799 tablet_service.cc:1467] Processing CreateTablet for tablet c276032387124d2aa28ce105a84a6b86 (DEFAULT_TABLE table=test-table [id=96e7fe24a9ab48b39cf342895a7e62e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:36:15.748981 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:15.729502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45523 (local address 127.0.0.1:51176)
0926 03:36:15.740184 (+ 10682us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:15.740204 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:15.740331 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:15.741202 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:15.741207 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:15.741236 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:36:15.741692 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:15.741705 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:15.744192 (+ 2487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:15.744198 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:15.746145 (+ 1947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:15.746159 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:15.748011 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:15.748018 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:15.748203 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:15.748247 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 03:36:15.748348 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 03:36:15.748508 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10543,"thread_start_us":10494,"threads_started":1}
I20260926 03:36:15.749389 18016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:15.730346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51176 (local address 127.0.97.1:45523)
0926 03:36:15.739947 (+ 9601us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:15.739963 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:15.740459 (+ 496us) server_negotiation.cc:407] Connection header received
0926 03:36:15.740610 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:15.740620 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:15.740956 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:15.741795 (+ 839us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:36:15.742085 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:15.744017 (+ 1932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:15.746327 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:15.747785 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:15.747940 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:15.748235 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:15.748420 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 03:36:15.748660 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9303,"thread_start_us":9070,"threads_started":1}
I20260926 03:36:15.749473 17799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c276032387124d2aa28ce105a84a6b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:36:15.750105 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:15.727134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37973 (local address 127.0.0.1:38798)
0926 03:36:15.728054 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:15.728118 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:15.728374 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:15.742243 (+ 13869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:15.742248 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:15.742278 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:36:15.742658 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:15.742671 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:15.744913 (+ 2242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:15.744918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:15.747049 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:15.747065 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:15.749178 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:15.749185 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:15.749387 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:15.749436 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 03:36:15.749514 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 03:36:15.749683 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":1313,"threads_started":1}
I20260926 03:36:15.751125 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:15.728005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38798 (local address 127.0.97.3:37973)
0926 03:36:15.741073 (+ 13068us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:15.741082 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:15.741118 (+ 36us) server_negotiation.cc:407] Connection header received
0926 03:36:15.741235 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:15.741242 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:15.741992 (+ 750us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:15.742256 (+ 264us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:36:15.742858 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:15.744729 (+ 1871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:15.747241 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:15.748937 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:15.749883 (+ 946us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:15.750150 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:15.750270 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 03:36:15.750473 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12864,"thread_start_us":13647,"threads_started":1}
I20260926 03:36:15.754609 17933 tablet_service.cc:1467] Processing CreateTablet for tablet c276032387124d2aa28ce105a84a6b86 (DEFAULT_TABLE table=test-table [id=96e7fe24a9ab48b39cf342895a7e62e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:36:15.754575 17665 tablet_service.cc:1467] Processing CreateTablet for tablet c276032387124d2aa28ce105a84a6b86 (DEFAULT_TABLE table=test-table [id=96e7fe24a9ab48b39cf342895a7e62e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:36:15.756114 17933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c276032387124d2aa28ce105a84a6b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:36:15.756135 17665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c276032387124d2aa28ce105a84a6b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:36:15.768236 18017 tablet_bootstrap.cc:492] T c276032387124d2aa28ce105a84a6b86 P a7a8da2dc172444882f70f43168fed87: Bootstrap starting.
I20260926 03:36:15.771904 18018 tablet_bootstrap.cc:492] T c276032387124d2aa28ce105a84a6b86 P c583e21944694ed480911d934709e269: Bootstrap starting.
I20260926 03:36:15.773397 18017 tablet_bootstrap.cc:654] T c276032387124d2aa28ce105a84a6b86 P a7a8da2dc172444882f70f43168fed87: Neither blocks nor log segments found. Creating new log.
I20260926 03:36:15.774719 18017 log.cc:824] T c276032387124d2aa28ce105a84a6b86 P a7a8da2dc172444882f70f43168fed87: Log is configured to *not* fsync() on all Append() calls
I20260926 03:36:15.775813 18019 tablet_bootstrap.cc:492] T c276032387124d2aa28ce105a84a6b86 P 17e665bbeeb94346820b3ea46678d64b: Bootstrap starting.
I20260926 03:36:15.776583 18018 tablet_bootstrap.cc:654] T c276032387124d2aa28ce105a84a6b86 P c583e21944694ed480911d934709e269: Neither blocks nor log segments found. Creating new log.
I20260926 03:36:15.778025 18018 log.cc:824] T c276032387124d2aa28ce105a84a6b86 P c583e21944694ed480911d934709e269: Log is configured to *not* fsync() on all Append() calls
I20260926 03:36:15.780185 18017 tablet_bootstrap.cc:492] T c276032387124d2aa28ce105a84a6b86 P a7a8da2dc172444882f70f43168fed87: No bootstrap required, opened a new log
I20260926 03:36:15.780515 18017 ts_tablet_manager.cc:1397] T c276032387124d2aa28ce105a84a6b86 P a7a8da2dc172444882f70f43168fed87: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260926 03:36:15.780772 18019 tablet_bootstrap.cc:654] T c276032387124d2aa28ce105a84a6b86 P 17e665bbeeb94346820b3ea46678d64b: Neither blocks nor log segments found. Creating new log.
I20260926 03:36:15.781440 18018 tablet_bootstrap.cc:492] T c276032387124d2aa28ce105a84a6b86 P c583e21944694ed480911d934709e269: No bootstrap required, opened a new log
I20260926 03:36:15.781713 18018 ts_tablet_manager.cc:1397] T c276032387124d2aa28ce105a84a6b86 P c583e21944694ed480911d934709e269: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 03:36:15.782121 18019 log.cc:824] T c276032387124d2aa28ce105a84a6b86 P 17e665bbeeb94346820b3ea46678d64b: Log is configured to *not* fsync() on all Append() calls
I20260926 03:36:15.785127 18019 tablet_bootstrap.cc:492] T c276032387124d2aa28ce105a84a6b86 P 17e665bbeeb94346820b3ea46678d64b: No bootstrap required, opened a new log
I20260926 03:36:15.785426 18019 ts_tablet_manager.cc:1397] T c276032387124d2aa28ce105a84a6b86 P 17e665bbeeb94346820b3ea46678d64b: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260926 03:36:15.792138 18017 raft_consensus.cc:348] T c276032387124d2aa28ce105a84a6b86 P a7a8da2dc172444882f70f43168fed87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c583e21944694ed480911d934709e269" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37973 } } peers { permanent_uuid: "a7a8da2dc172444882f70f43168fed87" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34907 } } peers { permanent_uuid: "17e665bbeeb94346820b3ea46678d64b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45523 } }
I20260926 03:36:15.792690 18017 raft_consensus.cc:374] T c276032387124d2aa28ce105a84a6b86 P a7a8da2dc172444882f70f43168fed87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:36:15.792812 18017 raft_consensus.cc:729] T c276032387124d2aa28ce105a84a6b86 P a7a8da2dc172444882f70f43168fed87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7a8da2dc172444882f70f43168fed87, State: Initialized, Role: FOLLOWER
I20260926 03:36:15.792654 18018 raft_consensus.cc:348] T c276032387124d2aa28ce105a84a6b86 P c583e21944694ed480911d934709e269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c583e21944694ed480911d934709e269" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37973 } } peers { permanent_uuid: "a7a8da2dc172444882f70f43168fed87" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34907 } } peers { permanent_uuid: "17e665bbeeb94346820b3ea46678d64b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45523 } }
I20260926 03:36:15.793097 18018 raft_consensus.cc:374] T c276032387124d2aa28ce105a84a6b86 P c583e21944694ed480911d934709e269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:36:15.793198 18018 raft_consensus.cc:729] T c276032387124d2aa28ce105a84a6b86 P c583e21944694ed480911d934709e269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c583e21944694ed480911d934709e269, State: Initialized, Role: FOLLOWER
I20260926 03:36:15.793567 18017 consensus_queue.cc:260] T c276032387124d2aa28ce105a84a6b86 P a7a8da2dc172444882f70f43168fed87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c583e21944694ed480911d934709e269" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37973 } } peers { permanent_uuid: "a7a8da2dc172444882f70f43168fed87" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34907 } } peers { permanent_uuid: "17e665bbeeb94346820b3ea46678d64b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45523 } }
I20260926 03:36:15.793771 18018 consensus_queue.cc:260] T c276032387124d2aa28ce105a84a6b86 P c583e21944694ed480911d934709e269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c583e21944694ed480911d934709e269" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37973 } } peers { permanent_uuid: "a7a8da2dc172444882f70f43168fed87" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34907 } } peers { permanent_uuid: "17e665bbeeb94346820b3ea46678d64b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45523 } }
I20260926 03:36:15.794930 17864 heartbeater.cc:503] Master 127.0.97.62:35481 was elected leader, sending a full tablet report...
I20260926 03:36:15.794633 18019 raft_consensus.cc:348] T c276032387124d2aa28ce105a84a6b86 P 17e665bbeeb94346820b3ea46678d64b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c583e21944694ed480911d934709e269" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37973 } } peers { permanent_uuid: "a7a8da2dc172444882f70f43168fed87" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34907 } } peers { permanent_uuid: "17e665bbeeb94346820b3ea46678d64b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45523 } }
I20260926 03:36:15.795528 18019 raft_consensus.cc:374] T c276032387124d2aa28ce105a84a6b86 P 17e665bbeeb94346820b3ea46678d64b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:36:15.795626 18019 raft_consensus.cc:729] T c276032387124d2aa28ce105a84a6b86 P 17e665bbeeb94346820b3ea46678d64b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17e665bbeeb94346820b3ea46678d64b, State: Initialized, Role: FOLLOWER
I20260926 03:36:15.795760 18017 ts_tablet_manager.cc:1428] T c276032387124d2aa28ce105a84a6b86 P a7a8da2dc172444882f70f43168fed87: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260926 03:36:15.796342 18019 consensus_queue.cc:260] T c276032387124d2aa28ce105a84a6b86 P 17e665bbeeb94346820b3ea46678d64b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c583e21944694ed480911d934709e269" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37973 } } peers { permanent_uuid: "a7a8da2dc172444882f70f43168fed87" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34907 } } peers { permanent_uuid: "17e665bbeeb94346820b3ea46678d64b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45523 } }
I20260926 03:36:15.797266 17998 heartbeater.cc:503] Master 127.0.97.62:35481 was elected leader, sending a full tablet report...
I20260926 03:36:15.797441 18018 ts_tablet_manager.cc:1428] T c276032387124d2aa28ce105a84a6b86 P c583e21944694ed480911d934709e269: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260926 03:36:15.799825 18019 ts_tablet_manager.cc:1428] T c276032387124d2aa28ce105a84a6b86 P 17e665bbeeb94346820b3ea46678d64b: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260926 03:36:15.849675 18025 raft_consensus.cc:482] T c276032387124d2aa28ce105a84a6b86 P 17e665bbeeb94346820b3ea46678d64b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:36:15.850129 18025 raft_consensus.cc:504] T c276032387124d2aa28ce105a84a6b86 P 17e665bbeeb94346820b3ea46678d64b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c583e21944694ed480911d934709e269" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37973 } } peers { permanent_uuid: "a7a8da2dc172444882f70f43168fed87" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34907 } } peers { permanent_uuid: "17e665bbeeb94346820b3ea46678d64b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45523 } }
I20260926 03:36:15.853039 18025 leader_election.cc:288] T c276032387124d2aa28ce105a84a6b86 P 17e665bbeeb94346820b3ea46678d64b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c583e21944694ed480911d934709e269 (127.0.97.3:37973), a7a8da2dc172444882f70f43168fed87 (127.0.97.2:34907)
I20260926 03:36:15.863588 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:15.853647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39465 (local address 127.0.97.2:34907)
0926 03:36:15.853894 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:15.853904 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:15.854786 (+ 882us) server_negotiation.cc:407] Connection header received
0926 03:36:15.855067 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:15.855075 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:15.855174 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:15.855327 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:36:15.856129 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:15.857480 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:15.860711 (+ 3231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:15.862265 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:15.862430 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:15.862572 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:15.863105 (+ 533us) server_negotiation.cc:299] Negotiation successful
0926 03:36:15.863268 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 03:36:15.863588 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:15.853498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50787 (local address 127.0.97.3:37973)
0926 03:36:15.853786 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:15.853793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:15.854832 (+ 1039us) server_negotiation.cc:407] Connection header received
0926 03:36:15.855009 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:15.855015 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:15.855119 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:15.855237 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:36:15.856128 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:15.857479 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:15.860765 (+ 3286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:15.862242 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:15.862430 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:15.862573 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:15.863105 (+ 532us) server_negotiation.cc:299] Negotiation successful
0926 03:36:15.863258 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 03:36:15.863945 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:15.853484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37973 (local address 127.0.97.1:50787)
0926 03:36:15.854696 (+ 1212us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:15.854713 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:15.854844 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:15.855285 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:15.855293 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:15.855328 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:36:15.855926 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:15.855941 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:15.857625 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:15.857631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:15.860501 (+ 2870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:15.860519 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:15.862590 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:15.862596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:15.862859 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:15.862951 (+ 92us) client_negotiation.cc:769] Sending connection context
0926 03:36:15.863057 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 03:36:15.863360 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1093,"mutex_wait_us":431,"spinlock_wait_cycles":43904,"thread_start_us":509,"threads_started":1}
I20260926 03:36:15.864032 18026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:15.853483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34907 (local address 127.0.97.1:39465)
0926 03:36:15.854614 (+ 1131us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:15.854639 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:15.854834 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:15.855338 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:15.855341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:15.855360 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:36:15.855926 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:15.855941 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:15.857627 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:15.857631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:15.860501 (+ 2870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:15.860519 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:15.862602 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:15.862607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:15.862853 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:15.862952 (+ 99us) client_negotiation.cc:769] Sending connection context
0926 03:36:15.863056 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 03:36:15.863360 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":967,"mutex_wait_us":277,"thread_start_us":577,"threads_started":1}
I20260926 03:36:15.865204 17953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c276032387124d2aa28ce105a84a6b86" candidate_uuid: "17e665bbeeb94346820b3ea46678d64b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c583e21944694ed480911d934709e269" is_pre_election: true
I20260926 03:36:15.865204 17819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c276032387124d2aa28ce105a84a6b86" candidate_uuid: "17e665bbeeb94346820b3ea46678d64b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7a8da2dc172444882f70f43168fed87" is_pre_election: true
I20260926 03:36:15.865835 17819 raft_consensus.cc:2454] T c276032387124d2aa28ce105a84a6b86 P a7a8da2dc172444882f70f43168fed87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17e665bbeeb94346820b3ea46678d64b in term 0.
I20260926 03:36:15.865834 17953 raft_consensus.cc:2454] T c276032387124d2aa28ce105a84a6b86 P c583e21944694ed480911d934709e269 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17e665bbeeb94346820b3ea46678d64b in term 0.
I20260926 03:36:15.866607 17621 leader_election.cc:302] T c276032387124d2aa28ce105a84a6b86 P 17e665bbeeb94346820b3ea46678d64b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 17e665bbeeb94346820b3ea46678d64b, a7a8da2dc172444882f70f43168fed87, c583e21944694ed480911d934709e269; no voters:
I20260926 03:36:15.867044 18025 raft_consensus.cc:2781] T c276032387124d2aa28ce105a84a6b86 P 17e665bbeeb94346820b3ea46678d64b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:36:15.867239 18025 raft_consensus.cc:482] T c276032387124d2aa28ce105a84a6b86 P 17e665bbeeb94346820b3ea46678d64b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:36:15.867378 18025 raft_consensus.cc:3037] T c276032387124d2aa28ce105a84a6b86 P 17e665bbeeb94346820b3ea46678d64b [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:36:15.870807 18025 raft_consensus.cc:504] T c276032387124d2aa28ce105a84a6b86 P 17e665bbeeb94346820b3ea46678d64b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c583e21944694ed480911d934709e269" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37973 } } peers { permanent_uuid: "a7a8da2dc172444882f70f43168fed87" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34907 } } peers { permanent_uuid: "17e665bbeeb94346820b3ea46678d64b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45523 } }
I20260926 03:36:15.871538 18025 leader_election.cc:288] T c276032387124d2aa28ce105a84a6b86 P 17e665bbeeb94346820b3ea46678d64b [CANDIDATE]: Term 1 election: Requested vote from peers c583e21944694ed480911d934709e269 (127.0.97.3:37973), a7a8da2dc172444882f70f43168fed87 (127.0.97.2:34907)
I20260926 03:36:15.872118 17953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c276032387124d2aa28ce105a84a6b86" candidate_uuid: "17e665bbeeb94346820b3ea46678d64b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c583e21944694ed480911d934709e269"
I20260926 03:36:15.872128 17819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c276032387124d2aa28ce105a84a6b86" candidate_uuid: "17e665bbeeb94346820b3ea46678d64b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7a8da2dc172444882f70f43168fed87"
I20260926 03:36:15.872364 17953 raft_consensus.cc:3037] T c276032387124d2aa28ce105a84a6b86 P c583e21944694ed480911d934709e269 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:36:15.872364 17819 raft_consensus.cc:3037] T c276032387124d2aa28ce105a84a6b86 P a7a8da2dc172444882f70f43168fed87 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:36:15.876227 17953 raft_consensus.cc:2454] T c276032387124d2aa28ce105a84a6b86 P c583e21944694ed480911d934709e269 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17e665bbeeb94346820b3ea46678d64b in term 1.
I20260926 03:36:15.876243 17819 raft_consensus.cc:2454] T c276032387124d2aa28ce105a84a6b86 P a7a8da2dc172444882f70f43168fed87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17e665bbeeb94346820b3ea46678d64b in term 1.
I20260926 03:36:15.876719 17621 leader_election.cc:302] T c276032387124d2aa28ce105a84a6b86 P 17e665bbeeb94346820b3ea46678d64b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17e665bbeeb94346820b3ea46678d64b, a7a8da2dc172444882f70f43168fed87; no voters:
I20260926 03:36:15.877045 18025 raft_consensus.cc:2781] T c276032387124d2aa28ce105a84a6b86 P 17e665bbeeb94346820b3ea46678d64b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:36:15.877581 18025 raft_consensus.cc:686] T c276032387124d2aa28ce105a84a6b86 P 17e665bbeeb94346820b3ea46678d64b [term 1 LEADER]: Becoming Leader. State: Replica: 17e665bbeeb94346820b3ea46678d64b, State: Running, Role: LEADER
I20260926 03:36:15.878149 18025 consensus_queue.cc:237] T c276032387124d2aa28ce105a84a6b86 P 17e665bbeeb94346820b3ea46678d64b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c583e21944694ed480911d934709e269" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37973 } } peers { permanent_uuid: "a7a8da2dc172444882f70f43168fed87" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34907 } } peers { permanent_uuid: "17e665bbeeb94346820b3ea46678d64b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45523 } }
I20260926 03:36:15.883828 17539 catalog_manager.cc:5495] T c276032387124d2aa28ce105a84a6b86 P 17e665bbeeb94346820b3ea46678d64b reported cstate change: term changed from 0 to 1, leader changed from <none> to 17e665bbeeb94346820b3ea46678d64b (127.0.97.1). New cstate: current_term: 1 leader_uuid: "17e665bbeeb94346820b3ea46678d64b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c583e21944694ed480911d934709e269" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7a8da2dc172444882f70f43168fed87" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17e665bbeeb94346820b3ea46678d64b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45523 } health_report { overall_health: HEALTHY } } }
W20260926 03:36:15.891901 17999 tablet.cc:2369] T c276032387124d2aa28ce105a84a6b86 P c583e21944694ed480911d934709e269: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:36:15.933506 18016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:15.927541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51180 (local address 127.0.97.1:45523)
0926 03:36:15.927825 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:15.927831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:15.927851 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:36:15.928084 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:15.928089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:15.928222 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:15.928366 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:36:15.928784 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:15.929617 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:15.931102 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:15.931640 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:15.931774 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:15.932210 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:36:15.932966 (+ 756us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:36:15.933088 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:15.933186 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 03:36:15.933272 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260926 03:36:15.943614 17819 raft_consensus.cc:1261] T c276032387124d2aa28ce105a84a6b86 P a7a8da2dc172444882f70f43168fed87 [term 1 FOLLOWER]: Refusing update from remote peer 17e665bbeeb94346820b3ea46678d64b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:36:15.943614 17953 raft_consensus.cc:1261] T c276032387124d2aa28ce105a84a6b86 P c583e21944694ed480911d934709e269 [term 1 FOLLOWER]: Refusing update from remote peer 17e665bbeeb94346820b3ea46678d64b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:36:15.944876 18028 consensus_queue.cc:1035] T c276032387124d2aa28ce105a84a6b86 P 17e665bbeeb94346820b3ea46678d64b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7a8da2dc172444882f70f43168fed87" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 03:36:15.945158 17731 tablet.cc:2369] T c276032387124d2aa28ce105a84a6b86 P 17e665bbeeb94346820b3ea46678d64b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:36:15.945358 18025 consensus_queue.cc:1035] T c276032387124d2aa28ce105a84a6b86 P 17e665bbeeb94346820b3ea46678d64b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c583e21944694ed480911d934709e269" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:36:15.960623 18033 mvcc.cc:205] Tried to move back new op lower bound from 7333452906251206656 to 7333452906001674240. Current Snapshot: MvccSnapshot[applied={T|T < 7333452906251206656}]
I20260926 03:36:15.975034 17539 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:36:15.975417 17539 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:36:15.979701 17539 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6700b06e9dd44bfca8f46f690649f3ef: Sending DeleteTablet for 3 replicas of tablet c276032387124d2aa28ce105a84a6b86
I20260926 03:36:15.981038 17933 tablet_service.cc:1514] Processing DeleteTablet for tablet c276032387124d2aa28ce105a84a6b86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:36:15 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38798
I20260926 03:36:15.981124 17799 tablet_service.cc:1514] Processing DeleteTablet for tablet c276032387124d2aa28ce105a84a6b86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:36:15 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60854
I20260926 03:36:15.981412 17665 tablet_service.cc:1514] Processing DeleteTablet for tablet c276032387124d2aa28ce105a84a6b86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:36:15 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51176
I20260926 03:36:15.984834 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 17602
I20260926 03:36:15.993872 18042 tablet_replica.cc:330] T c276032387124d2aa28ce105a84a6b86 P c583e21944694ed480911d934709e269: stopping tablet replica
I20260926 03:36:15.994567 18042 raft_consensus.cc:2229] T c276032387124d2aa28ce105a84a6b86 P c583e21944694ed480911d934709e269 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:36:15.995327 18042 raft_consensus.cc:2258] T c276032387124d2aa28ce105a84a6b86 P c583e21944694ed480911d934709e269 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:36:15.997174 18044 tablet_replica.cc:330] T c276032387124d2aa28ce105a84a6b86 P 17e665bbeeb94346820b3ea46678d64b: stopping tablet replica
I20260926 03:36:15.997641 18044 raft_consensus.cc:2229] T c276032387124d2aa28ce105a84a6b86 P 17e665bbeeb94346820b3ea46678d64b [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:36:15.998454 18044 raft_consensus.cc:2258] T c276032387124d2aa28ce105a84a6b86 P 17e665bbeeb94346820b3ea46678d64b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:36:15.999262 18043 tablet_replica.cc:330] T c276032387124d2aa28ce105a84a6b86 P a7a8da2dc172444882f70f43168fed87: stopping tablet replica
I20260926 03:36:15.999727 18043 raft_consensus.cc:2229] T c276032387124d2aa28ce105a84a6b86 P a7a8da2dc172444882f70f43168fed87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:36:16.000310 18043 raft_consensus.cc:2258] T c276032387124d2aa28ce105a84a6b86 P a7a8da2dc172444882f70f43168fed87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:36:16.001420 18042 ts_tablet_manager.cc:1905] T c276032387124d2aa28ce105a84a6b86 P c583e21944694ed480911d934709e269: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:36:16.002393 18043 ts_tablet_manager.cc:1905] T c276032387124d2aa28ce105a84a6b86 P a7a8da2dc172444882f70f43168fed87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:36:16.009078 18042 ts_tablet_manager.cc:1918] T c276032387124d2aa28ce105a84a6b86 P c583e21944694ed480911d934709e269: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:36:16.009239 18042 log.cc:1196] T c276032387124d2aa28ce105a84a6b86 P c583e21944694ed480911d934709e269: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-2/wal/wals/c276032387124d2aa28ce105a84a6b86
I20260926 03:36:16.009781 18042 ts_tablet_manager.cc:1939] T c276032387124d2aa28ce105a84a6b86 P c583e21944694ed480911d934709e269: Deleting consensus metadata
I20260926 03:36:16.010242 18043 ts_tablet_manager.cc:1918] T c276032387124d2aa28ce105a84a6b86 P a7a8da2dc172444882f70f43168fed87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:36:16.010412 18043 log.cc:1196] T c276032387124d2aa28ce105a84a6b86 P a7a8da2dc172444882f70f43168fed87: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-1/wal/wals/c276032387124d2aa28ce105a84a6b86
I20260926 03:36:16.010963 18043 ts_tablet_manager.cc:1939] T c276032387124d2aa28ce105a84a6b86 P a7a8da2dc172444882f70f43168fed87: Deleting consensus metadata
I20260926 03:36:16.012058 17526 catalog_manager.cc:4841] TS c583e21944694ed480911d934709e269 (127.0.97.3:37973): tablet c276032387124d2aa28ce105a84a6b86 (table test-table [id=96e7fe24a9ab48b39cf342895a7e62e1]) successfully deleted
I20260926 03:36:16.012734 17528 catalog_manager.cc:4841] TS a7a8da2dc172444882f70f43168fed87 (127.0.97.2:34907): tablet c276032387124d2aa28ce105a84a6b86 (table test-table [id=96e7fe24a9ab48b39cf342895a7e62e1]) successfully deleted
I20260926 03:36:16.012846 18044 ts_tablet_manager.cc:1905] T c276032387124d2aa28ce105a84a6b86 P 17e665bbeeb94346820b3ea46678d64b: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 03:36:16 dist-test-slave-dt7p krb5kdc[17496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:36:16.021132 18044 ts_tablet_manager.cc:1918] T c276032387124d2aa28ce105a84a6b86 P 17e665bbeeb94346820b3ea46678d64b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:36:16.021332 18044 log.cc:1196] T c276032387124d2aa28ce105a84a6b86 P 17e665bbeeb94346820b3ea46678d64b: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790393463219689-388-0/minicluster-data/ts-0/wal/wals/c276032387124d2aa28ce105a84a6b86
I20260926 03:36:16.021898 18044 ts_tablet_manager.cc:1939] T c276032387124d2aa28ce105a84a6b86 P 17e665bbeeb94346820b3ea46678d64b: Deleting consensus metadata
I20260926 03:36:16.022336 18016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:15.986335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51192 (local address 127.0.97.1:45523)
0926 03:36:15.986585 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:15.986595 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:15.996416 (+ 9821us) server_negotiation.cc:407] Connection header received
0926 03:36:15.997007 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:15.997014 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:15.997122 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:15.997288 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:15.999492 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:16.000838 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:16.006666 (+ 5828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:16.007441 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:16.007546 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:16.015261 (+ 7715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:16.015274 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:16.015290 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:36:16.015340 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:16.019119 (+ 3779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:16.019856 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:16.019876 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:16.019880 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:16.020032 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:16.020431 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:16.020441 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:16.020443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:16.020894 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:16.021349 (+ 455us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:16.021583 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 03:36:16.021853 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 03:36:16.023078 17725 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:36:16.023809 17528 catalog_manager.cc:4841] TS 17e665bbeeb94346820b3ea46678d64b (127.0.97.1:45523): tablet c276032387124d2aa28ce105a84a6b86 (table test-table [id=96e7fe24a9ab48b39cf342895a7e62e1]) successfully deleted
I20260926 03:36:16.148470 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 17602
I20260926 03:36:16.170274 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 17736
Sep 26 03:36:16 dist-test-slave-dt7p krb5kdc[17496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:36:16.187252 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:16.171166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60870 (local address 127.0.97.2:34907)
0926 03:36:16.171372 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:16.171382 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:16.171472 (+ 90us) server_negotiation.cc:407] Connection header received
0926 03:36:16.171727 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:16.171732 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:16.171835 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:16.171955 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:16.173509 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:16.174359 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:16.175753 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:16.176171 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:16.176311 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:16.181072 (+ 4761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:16.181091 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:16.181112 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:36:16.181151 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:16.184077 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:16.184887 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:16.184896 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:16.184898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:16.185033 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:16.185438 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:16.185448 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:16.185450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:16.185903 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:16.186257 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:16.186556 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 03:36:16.186827 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:36:16.188053 17859 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:36:16.285439 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 17736
I20260926 03:36:16.305653 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 17870
Sep 26 03:36:16 dist-test-slave-dt7p krb5kdc[17496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:36:16.321877 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:16.306574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38804 (local address 127.0.97.3:37973)
0926 03:36:16.306774 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:16.306779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:16.306800 (+ 21us) server_negotiation.cc:407] Connection header received
0926 03:36:16.306981 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:16.306986 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:16.307074 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:16.307188 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:16.308697 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:16.309535 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:16.310812 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:16.311240 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:16.311359 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:16.315982 (+ 4623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:16.316000 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:16.316021 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:36:16.316058 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:16.319088 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:16.319808 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:16.319817 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:16.319819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:16.319957 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:16.320281 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:16.320284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:16.320286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:16.320745 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:16.321009 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:16.321260 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 03:36:16.321449 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 03:36:16.322634 17993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:36:16.419396 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 17870
I20260926 03:36:16.438441 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 17509
I20260926 03:36:16.439885 17571 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:36:16.568848 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 17509
2026-09-26T03:36:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4830 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T03:36:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:36:16Z Disabled control of system clock
I20260926 03:36:16.658565 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38193
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 03:36:17.052666 18053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:36:17.053089 18053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:36:17.053185 18053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:36:17.061213 18053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:36:17.061331 18053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:36:17.061371 18053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:36:17.061431 18053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:36:17.061478 18053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:36:17.071220 18053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38193
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:36:17.072867 18053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:36:17.075002 18053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:36:17.087869 18061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:17.087838 18058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:17.087918 18059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:36:17.088675 18053 server_base.cc:1034] running on GCE node
I20260926 03:36:17.089537 18053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:36:17.091025 18053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:36:17.092267 18053 hybrid_clock.cc:648] HybridClock initialized: now 1790393777092218 us; error 48 us; skew 500 ppm
I20260926 03:36:17.092726 18053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:36:17.101676 18053 webserver.cc:460] Webserver started at http://127.0.97.62:39021/ using document root <none> and password file <none>
I20260926 03:36:17.102574 18053 fs_manager.cc:359] Metadata directory not provided
I20260926 03:36:17.102720 18053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:36:17.103109 18053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:36:17.106585 18053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "925b3f50fe904e478d6f1ceecb656177"
format_stamp: "Formatted at 2026-09-26 03:36:17 on dist-test-slave-dt7p"
server_key: "f1b16a037243664b8c1e48d357330e8e"
server_key_iv: "6819cda23a6d30e937978ad3d0897551"
server_key_version: "encryptionkey@0"
I20260926 03:36:17.107458 18053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "925b3f50fe904e478d6f1ceecb656177"
format_stamp: "Formatted at 2026-09-26 03:36:17 on dist-test-slave-dt7p"
server_key: "f1b16a037243664b8c1e48d357330e8e"
server_key_iv: "6819cda23a6d30e937978ad3d0897551"
server_key_version: "encryptionkey@0"
I20260926 03:36:17.113356 18053 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260926 03:36:17.117244 18067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:17.118762 18053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:36:17.118971 18053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "925b3f50fe904e478d6f1ceecb656177"
format_stamp: "Formatted at 2026-09-26 03:36:17 on dist-test-slave-dt7p"
server_key: "f1b16a037243664b8c1e48d357330e8e"
server_key_iv: "6819cda23a6d30e937978ad3d0897551"
server_key_version: "encryptionkey@0"
I20260926 03:36:17.119208 18053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:36:17.133199 18053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:36:17.134174 18053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:36:17.134511 18053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:36:17.155934 18053 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38193
I20260926 03:36:17.155961 18118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38193 every 8 connection(s)
I20260926 03:36:17.157778 18053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:36:17.162037 18119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:36:17.166450 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 18053
I20260926 03:36:17.166647 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:36:17.167115 388 external_mini_cluster.cc:1428] Setting key db9b402958694c61a63462f97d1924a4
I20260926 03:36:17.174029 18119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 925b3f50fe904e478d6f1ceecb656177: Bootstrap starting.
I20260926 03:36:17.177501 18119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 925b3f50fe904e478d6f1ceecb656177: Neither blocks nor log segments found. Creating new log.
I20260926 03:36:17.178656 18119 log.cc:824] T 00000000000000000000000000000000 P 925b3f50fe904e478d6f1ceecb656177: Log is configured to *not* fsync() on all Append() calls
I20260926 03:36:17.182279 18119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 925b3f50fe904e478d6f1ceecb656177: No bootstrap required, opened a new log
I20260926 03:36:17.182492 18122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:17.169148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34896 (local address 127.0.97.62:38193)
0926 03:36:17.170205 (+ 1057us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:17.170226 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:17.170271 (+ 45us) server_negotiation.cc:407] Connection header received
0926 03:36:17.172659 (+ 2388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:17.172684 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:17.173182 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:17.173582 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:17.174205 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:17.175586 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:17.177161 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:17.177995 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:17.178197 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:17.179552 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:17.179562 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:17.179575 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:36:17.180267 (+ 692us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:17.180383 (+ 116us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:36:17.180410 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:17.180529 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:17.180803 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 03:36:17.181151 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":376,"threads_started":1}
I20260926 03:36:17.191385 18119 raft_consensus.cc:348] T 00000000000000000000000000000000 P 925b3f50fe904e478d6f1ceecb656177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "925b3f50fe904e478d6f1ceecb656177" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38193 } }
I20260926 03:36:17.191845 18119 raft_consensus.cc:374] T 00000000000000000000000000000000 P 925b3f50fe904e478d6f1ceecb656177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:36:17.191924 18119 raft_consensus.cc:729] T 00000000000000000000000000000000 P 925b3f50fe904e478d6f1ceecb656177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 925b3f50fe904e478d6f1ceecb656177, State: Initialized, Role: FOLLOWER
I20260926 03:36:17.192519 18119 consensus_queue.cc:260] T 00000000000000000000000000000000 P 925b3f50fe904e478d6f1ceecb656177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "925b3f50fe904e478d6f1ceecb656177" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38193 } }
I20260926 03:36:17.192905 18119 raft_consensus.cc:388] T 00000000000000000000000000000000 P 925b3f50fe904e478d6f1ceecb656177 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:36:17.193069 18119 raft_consensus.cc:482] T 00000000000000000000000000000000 P 925b3f50fe904e478d6f1ceecb656177 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:36:17.193331 18119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 925b3f50fe904e478d6f1ceecb656177 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:36:17.195760 18119 raft_consensus.cc:504] T 00000000000000000000000000000000 P 925b3f50fe904e478d6f1ceecb656177 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "925b3f50fe904e478d6f1ceecb656177" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38193 } }
I20260926 03:36:17.196344 18119 leader_election.cc:302] T 00000000000000000000000000000000 P 925b3f50fe904e478d6f1ceecb656177 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 925b3f50fe904e478d6f1ceecb656177; no voters:
I20260926 03:36:17.197134 18119 leader_election.cc:288] T 00000000000000000000000000000000 P 925b3f50fe904e478d6f1ceecb656177 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:36:17.197310 18124 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 925b3f50fe904e478d6f1ceecb656177 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:36:17.198148 18124 raft_consensus.cc:686] T 00000000000000000000000000000000 P 925b3f50fe904e478d6f1ceecb656177 [term 1 LEADER]: Becoming Leader. State: Replica: 925b3f50fe904e478d6f1ceecb656177, State: Running, Role: LEADER
I20260926 03:36:17.198789 18124 consensus_queue.cc:237] T 00000000000000000000000000000000 P 925b3f50fe904e478d6f1ceecb656177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "925b3f50fe904e478d6f1ceecb656177" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38193 } }
I20260926 03:36:17.199604 18119 sys_catalog.cc:564] T 00000000000000000000000000000000 P 925b3f50fe904e478d6f1ceecb656177 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:36:17.202056 18125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 925b3f50fe904e478d6f1ceecb656177 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "925b3f50fe904e478d6f1ceecb656177" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "925b3f50fe904e478d6f1ceecb656177" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38193 } } }
I20260926 03:36:17.202507 18125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 925b3f50fe904e478d6f1ceecb656177 [sys.catalog]: This master's current role is: LEADER
I20260926 03:36:17.202828 18126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 925b3f50fe904e478d6f1ceecb656177 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 925b3f50fe904e478d6f1ceecb656177. Latest consensus state: current_term: 1 leader_uuid: "925b3f50fe904e478d6f1ceecb656177" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "925b3f50fe904e478d6f1ceecb656177" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38193 } } }
I20260926 03:36:17.203135 18126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 925b3f50fe904e478d6f1ceecb656177 [sys.catalog]: This master's current role is: LEADER
I20260926 03:36:17.205052 18132 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:36:17.210427 18132 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:36:17.220731 18132 catalog_manager.cc:1347] Generated new cluster ID: e6961dad9e5d4f408055fd2bb337d95d
I20260926 03:36:17.220885 18132 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:36:17.253027 18132 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:36:17.254552 18132 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:36:17.261368 18132 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 925b3f50fe904e478d6f1ceecb656177: Generated new TSK 0
I20260926 03:36:17.262286 18132 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:36:17.275238 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38193
--builtin_ntp_servers=127.0.97.20:35563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 03:36:17.660324 18143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:36:17.660689 18143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:36:17.660862 18143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:36:17.668530 18143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:36:17.668707 18143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:36:17.668901 18143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:36:17.678474 18143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:36:17.680243 18143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:36:17.682469 18143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:36:17.698129 18149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:17.698164 18151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:17.698046 18148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:36:17.699164 18143 server_base.cc:1034] running on GCE node
I20260926 03:36:17.699816 18143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:36:17.701093 18143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:36:17.702337 18143 hybrid_clock.cc:648] HybridClock initialized: now 1790393777702271 us; error 51 us; skew 500 ppm
I20260926 03:36:17.702732 18143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:36:17.705592 18143 webserver.cc:460] Webserver started at http://127.0.97.1:45731/ using document root <none> and password file <none>
I20260926 03:36:17.706512 18143 fs_manager.cc:359] Metadata directory not provided
I20260926 03:36:17.706641 18143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:36:17.707017 18143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:36:17.710383 18143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-0/data/instance:
uuid: "f3d2f65dba1e4a6a91f451b644a806f6"
format_stamp: "Formatted at 2026-09-26 03:36:17 on dist-test-slave-dt7p"
server_key: "42d05a6cd46a02f3c5f74dbb0b0a7dc9"
server_key_iv: "2c336f338263aa7885d03d5dba2cac04"
server_key_version: "encryptionkey@0"
I20260926 03:36:17.711288 18143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f3d2f65dba1e4a6a91f451b644a806f6"
format_stamp: "Formatted at 2026-09-26 03:36:17 on dist-test-slave-dt7p"
server_key: "42d05a6cd46a02f3c5f74dbb0b0a7dc9"
server_key_iv: "2c336f338263aa7885d03d5dba2cac04"
server_key_version: "encryptionkey@0"
I20260926 03:36:17.717145 18143 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260926 03:36:17.721055 18157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:17.722544 18143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:36:17.722734 18143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-0/wal
uuid: "f3d2f65dba1e4a6a91f451b644a806f6"
format_stamp: "Formatted at 2026-09-26 03:36:17 on dist-test-slave-dt7p"
server_key: "42d05a6cd46a02f3c5f74dbb0b0a7dc9"
server_key_iv: "2c336f338263aa7885d03d5dba2cac04"
server_key_version: "encryptionkey@0"
I20260926 03:36:17.722986 18143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:36:17.744556 18143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:36:17.745551 18143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:36:17.745868 18143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:36:17.746922 18143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:36:17.748629 18143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:36:17.748752 18143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:17.748893 18143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:36:17.748988 18143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:17.785429 18143 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44961
I20260926 03:36:17.785498 18269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44961 every 8 connection(s)
I20260926 03:36:17.786974 18143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:36:17.788882 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 18143
I20260926 03:36:17.789047 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:36:17.789517 388 external_mini_cluster.cc:1428] Setting key 68fa7046fe4028d9efdd6791212057e3
I20260926 03:36:17.792980 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38193
--builtin_ntp_servers=127.0.97.20:35563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 03:36:17.805663 18272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:17.791684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50539 (local address 127.0.97.62:38193)
0926 03:36:17.792269 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:17.792277 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:17.794019 (+ 1742us) server_negotiation.cc:407] Connection header received
0926 03:36:17.797122 (+ 3103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:17.797127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:17.797209 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:17.797321 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:17.798777 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:17.799630 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:17.802196 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:17.802649 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:17.802762 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:17.804515 (+ 1753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:17.804520 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:17.804523 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:36:17.804590 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:17.804661 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:36:17.804685 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:17.804796 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:17.805026 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 03:36:17.805164 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":306,"threads_started":1}
I20260926 03:36:17.806707 18273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:17.792522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38193 (local address 127.0.97.1:50539)
0926 03:36:17.793816 (+ 1294us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:17.793870 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:17.796819 (+ 2949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:17.797587 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:17.797601 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:17.797725 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:36:17.798596 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:17.798624 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:17.799782 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:17.799791 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:17.802055 (+ 2264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:17.802074 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:17.802886 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:17.802894 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:17.803241 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:17.804111 (+ 870us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:36:17.804117 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:36:17.804180 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:36:17.804307 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:36:17.804792 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:36:17.804800 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:36:17.804806 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 03:36:17.805000 (+ 194us) client_negotiation.cc:240] Negotiation successful
0926 03:36:17.805395 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":355,"threads_started":1}
I20260926 03:36:17.808394 18270 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38193
I20260926 03:36:17.808811 18270 heartbeater.cc:464] Registering TS with master...
I20260926 03:36:17.809849 18270 heartbeater.cc:511] Master 127.0.97.62:38193 requested a full tablet report, sending...
I20260926 03:36:17.812373 18084 ts_manager.cc:194] Registered new tserver with Master: f3d2f65dba1e4a6a91f451b644a806f6 (127.0.97.1:44961)
I20260926 03:36:17.814070 18084 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:50539
W20260926 03:36:18.183544 18274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:36:18.183871 18274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:36:18.184026 18274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:36:18.191939 18274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:36:18.192070 18274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:36:18.192272 18274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:36:18.202658 18274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:36:18.204499 18274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:36:18.206833 18274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:36:18.221999 18279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:18.221911 18280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:18.221951 18282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:36:18.223495 18274 server_base.cc:1034] running on GCE node
I20260926 03:36:18.224224 18274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:36:18.225531 18274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:36:18.226920 18274 hybrid_clock.cc:648] HybridClock initialized: now 1790393778226873 us; error 37 us; skew 500 ppm
I20260926 03:36:18.227308 18274 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:36:18.229998 18274 webserver.cc:460] Webserver started at http://127.0.97.2:38843/ using document root <none> and password file <none>
I20260926 03:36:18.230813 18274 fs_manager.cc:359] Metadata directory not provided
I20260926 03:36:18.230907 18274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:36:18.231271 18274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:36:18.234658 18274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-1/data/instance:
uuid: "91b5a5178aac4470b13495d5643597e9"
format_stamp: "Formatted at 2026-09-26 03:36:18 on dist-test-slave-dt7p"
server_key: "5762a1dd7cc7178a9c3d5870ba6ec0a7"
server_key_iv: "5443e51b7f5cb7a7b89780f8f5cbdd7c"
server_key_version: "encryptionkey@0"
I20260926 03:36:18.235550 18274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-1/wal/instance:
uuid: "91b5a5178aac4470b13495d5643597e9"
format_stamp: "Formatted at 2026-09-26 03:36:18 on dist-test-slave-dt7p"
server_key: "5762a1dd7cc7178a9c3d5870ba6ec0a7"
server_key_iv: "5443e51b7f5cb7a7b89780f8f5cbdd7c"
server_key_version: "encryptionkey@0"
I20260926 03:36:18.241379 18274 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 03:36:18.245079 18288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:18.246496 18274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:36:18.246681 18274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-1/wal
uuid: "91b5a5178aac4470b13495d5643597e9"
format_stamp: "Formatted at 2026-09-26 03:36:18 on dist-test-slave-dt7p"
server_key: "5762a1dd7cc7178a9c3d5870ba6ec0a7"
server_key_iv: "5443e51b7f5cb7a7b89780f8f5cbdd7c"
server_key_version: "encryptionkey@0"
I20260926 03:36:18.246914 18274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:36:18.275046 18274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:36:18.276089 18274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:36:18.276433 18274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:36:18.277670 18274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:36:18.279425 18274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:36:18.279562 18274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:18.279695 18274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:36:18.279789 18274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:18.317234 18274 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45503
I20260926 03:36:18.317337 18400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45503 every 8 connection(s)
I20260926 03:36:18.318819 18274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:36:18.328805 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 18274
I20260926 03:36:18.329016 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:36:18.329501 388 external_mini_cluster.cc:1428] Setting key 7d488bf756ed3da0b617725a9044ea8d
I20260926 03:36:18.332819 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38193
--builtin_ntp_servers=127.0.97.20:35563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 03:36:18.334623 18404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:18.322482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38193 (local address 127.0.97.2:44897)
0926 03:36:18.323632 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:18.323699 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:18.325705 (+ 2006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:18.326530 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:18.326540 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:18.326717 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:36:18.327360 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:18.327378 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:18.328650 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:18.328654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:18.330181 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:18.330199 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:18.331080 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:18.331087 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:18.331388 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:18.332199 (+ 811us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:36:18.332201 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:36:18.332248 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:36:18.332343 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:36:18.332815 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:36:18.332827 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:36:18.332838 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 03:36:18.333050 (+ 212us) client_negotiation.cc:240] Negotiation successful
0926 03:36:18.333461 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":346,"threads_started":1}
I20260926 03:36:18.335474 18403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:18.321795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44897 (local address 127.0.97.62:38193)
0926 03:36:18.322405 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:18.322410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:18.323833 (+ 1423us) server_negotiation.cc:407] Connection header received
0926 03:36:18.326006 (+ 2173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:18.326012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:18.326142 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:18.326285 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:18.327540 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:18.328466 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:18.330345 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:18.330821 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:18.330963 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:18.332611 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:18.332619 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:18.332624 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:36:18.332655 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:18.332704 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:36:18.332719 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:18.332823 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:18.335058 (+ 2235us) server_negotiation.cc:299] Negotiation successful
0926 03:36:18.335179 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":309,"threads_started":1}
I20260926 03:36:18.336845 18401 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38193
I20260926 03:36:18.337203 18401 heartbeater.cc:464] Registering TS with master...
I20260926 03:36:18.338078 18401 heartbeater.cc:511] Master 127.0.97.62:38193 requested a full tablet report, sending...
I20260926 03:36:18.339900 18084 ts_manager.cc:194] Registered new tserver with Master: 91b5a5178aac4470b13495d5643597e9 (127.0.97.2:45503)
I20260926 03:36:18.341259 18084 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:44897
W20260926 03:36:18.710695 18405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:36:18.711017 18405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:36:18.711112 18405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:36:18.718626 18405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:36:18.718734 18405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:36:18.718850 18405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:36:18.728940 18405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:36:18.730408 18405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:36:18.732432 18405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:36:18.746888 18411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:18.746929 18413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:18.746845 18410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:36:18.747995 18405 server_base.cc:1034] running on GCE node
I20260926 03:36:18.748746 18405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:36:18.749828 18405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:36:18.751220 18405 hybrid_clock.cc:648] HybridClock initialized: now 1790393778751179 us; error 60 us; skew 500 ppm
I20260926 03:36:18.751596 18405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:36:18.754173 18405 webserver.cc:460] Webserver started at http://127.0.97.3:42795/ using document root <none> and password file <none>
I20260926 03:36:18.754997 18405 fs_manager.cc:359] Metadata directory not provided
I20260926 03:36:18.755095 18405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:36:18.755419 18405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:36:18.758845 18405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-2/data/instance:
uuid: "a33971c3b9044bbcad6977c3d85e7e40"
format_stamp: "Formatted at 2026-09-26 03:36:18 on dist-test-slave-dt7p"
server_key: "35c8f4b9e67bcc5e3925114164940405"
server_key_iv: "d520b9700bab9ad58777041415baeaa2"
server_key_version: "encryptionkey@0"
I20260926 03:36:18.759701 18405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a33971c3b9044bbcad6977c3d85e7e40"
format_stamp: "Formatted at 2026-09-26 03:36:18 on dist-test-slave-dt7p"
server_key: "35c8f4b9e67bcc5e3925114164940405"
server_key_iv: "d520b9700bab9ad58777041415baeaa2"
server_key_version: "encryptionkey@0"
I20260926 03:36:18.765705 18405 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260926 03:36:18.769619 18419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:18.771106 18405 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260926 03:36:18.771294 18405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-2/wal
uuid: "a33971c3b9044bbcad6977c3d85e7e40"
format_stamp: "Formatted at 2026-09-26 03:36:18 on dist-test-slave-dt7p"
server_key: "35c8f4b9e67bcc5e3925114164940405"
server_key_iv: "d520b9700bab9ad58777041415baeaa2"
server_key_version: "encryptionkey@0"
I20260926 03:36:18.771498 18405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:36:18.787343 18405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:36:18.788364 18405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:36:18.788726 18405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:36:18.789884 18405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:36:18.791589 18405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:36:18.791683 18405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:18.791843 18405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:36:18.791935 18405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:18.818163 18270 heartbeater.cc:503] Master 127.0.97.62:38193 was elected leader, sending a full tablet report...
I20260926 03:36:18.828130 18405 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38915
I20260926 03:36:18.828179 18531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38915 every 8 connection(s)
I20260926 03:36:18.829749 18405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:36:18.832784 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 18405
I20260926 03:36:18.833001 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:36:18.838908 388 external_mini_cluster.cc:1428] Setting key 1fe2de93cc51e674130f3b6b4ebe2e2f
I20260926 03:36:18.846565 18403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:18.832863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54553 (local address 127.0.97.62:38193)
0926 03:36:18.833146 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:18.833156 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:18.835181 (+ 2025us) server_negotiation.cc:407] Connection header received
0926 03:36:18.838530 (+ 3349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:18.838535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:18.838633 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:18.838739 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:18.840283 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:18.841345 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:18.843521 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:18.844011 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:18.844131 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:18.845743 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:18.845749 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:18.845752 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:36:18.845775 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:18.845826 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:36:18.845839 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:18.845921 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:18.846168 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 03:36:18.846273 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 03:36:18.847729 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:18.833488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38193 (local address 127.0.97.3:54553)
0926 03:36:18.834916 (+ 1428us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:18.834994 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:18.838087 (+ 3093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:18.839092 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:18.839105 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:18.839252 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:36:18.840062 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:18.840083 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:18.841523 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:18.841529 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:18.843367 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:18.843381 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:18.844241 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:18.844246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:18.844489 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:18.845390 (+ 901us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:36:18.845393 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:36:18.845442 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:36:18.845540 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:36:18.845930 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:36:18.845938 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:36:18.845945 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 03:36:18.846105 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 03:36:18.846482 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":378,"threads_started":1}
I20260926 03:36:18.849404 18532 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38193
I20260926 03:36:18.849745 18532 heartbeater.cc:464] Registering TS with master...
I20260926 03:36:18.850721 18532 heartbeater.cc:511] Master 127.0.97.62:38193 requested a full tablet report, sending...
I20260926 03:36:18.852471 18084 ts_manager.cc:194] Registered new tserver with Master: a33971c3b9044bbcad6977c3d85e7e40 (127.0.97.3:38915)
I20260926 03:36:18.853576 18084 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:54553
I20260926 03:36:18.855973 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:36:18.867827 18403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:18.861193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34900 (local address 127.0.97.62:38193)
0926 03:36:18.861482 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:18.861492 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:18.861886 (+ 394us) server_negotiation.cc:407] Connection header received
0926 03:36:18.862068 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:18.862072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:18.862152 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:18.862264 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:18.862810 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:18.863639 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:18.865361 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:18.865831 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:18.865942 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:18.867023 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:18.867028 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:18.867031 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:36:18.867051 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:18.867088 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:36:18.867101 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:18.867201 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:18.867315 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 03:36:18.867409 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 03:36:18.872246 18084 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:36:18.879884 18084 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:36:18.910501 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:18.894129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45503 (local address 127.0.0.1:34158)
0926 03:36:18.899259 (+ 5130us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:18.899323 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:18.899626 (+ 303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:18.901225 (+ 1599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:18.901239 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:18.901289 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:36:18.901944 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:18.901972 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:18.904175 (+ 2203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:18.904184 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:18.907117 (+ 2933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:18.907141 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:18.909339 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:18.909348 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:18.909566 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:18.909617 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 03:36:18.909770 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 03:36:18.909984 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4731,"mutex_wait_us":59,"thread_start_us":7317,"threads_started":1}
I20260926 03:36:18.911299 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:18.894689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35124 (local address 127.0.97.1:44961)
0926 03:36:18.901242 (+ 6553us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:18.901252 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:18.902083 (+ 831us) server_negotiation.cc:407] Connection header received
0926 03:36:18.902216 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:18.902226 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:18.902410 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:18.902654 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:36:18.903718 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:18.905659 (+ 1941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:18.908251 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:18.909395 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:18.909533 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:18.909802 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:18.910309 (+ 507us) server_negotiation.cc:299] Negotiation successful
0926 03:36:18.910544 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6351,"thread_start_us":6175,"threads_started":1}
I20260926 03:36:18.911338 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:18.894713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34158 (local address 127.0.97.2:45503)
0926 03:36:18.898399 (+ 3686us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:18.898415 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:18.899879 (+ 1464us) server_negotiation.cc:407] Connection header received
0926 03:36:18.900054 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:18.900067 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:18.900453 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:18.900754 (+ 301us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:36:18.902233 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:18.904013 (+ 1780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:18.907371 (+ 3358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:18.909120 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:18.909280 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:18.909589 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:18.910827 (+ 1238us) server_negotiation.cc:299] Negotiation successful
0926 03:36:18.911004 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3410,"thread_start_us":3220,"threads_started":1}
I20260926 03:36:18.912107 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:18.903042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38915 (local address 127.0.0.1:51576)
0926 03:36:18.903772 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:18.903792 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:18.903923 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:18.904500 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:18.904505 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:18.904533 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:36:18.905214 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:18.905227 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:18.907141 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:18.907144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:18.909452 (+ 2308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:18.909472 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:18.911330 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:18.911337 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:18.911515 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:18.911551 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 03:36:18.911631 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 03:36:18.911748 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":119,"thread_start_us":402,"threads_started":1}
I20260926 03:36:18.912314 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:18.902308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51576 (local address 127.0.97.3:38915)
0926 03:36:18.903042 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:18.903051 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:18.903910 (+ 859us) server_negotiation.cc:407] Connection header received
0926 03:36:18.904087 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:18.904097 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:18.904294 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:18.904512 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:36:18.905479 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:18.906962 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:18.909742 (+ 2780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:18.911119 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:18.911282 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:18.911583 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:18.911681 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 03:36:18.911854 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":375,"threads_started":1}
I20260926 03:36:18.910831 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:18.894159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44961 (local address 127.0.0.1:35124)
0926 03:36:18.901758 (+ 7599us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:18.901778 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:18.901909 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:18.902955 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:18.902960 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:18.902984 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:36:18.903474 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:18.903491 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:18.905872 (+ 2381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:18.905877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:18.908045 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:18.908057 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:18.909922 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:18.909928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:18.910072 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:18.910115 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 03:36:18.910214 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 03:36:18.910355 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7466,"mutex_wait_us":10,"thread_start_us":4507,"threads_started":1}
I20260926 03:36:18.915915 18467 tablet_service.cc:1467] Processing CreateTablet for tablet 0254a3f73ab14460ba98658fbdf1e885 (DEFAULT_TABLE table=test-table [id=b69be1362a04450b966eb23f2b7f0b4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:36:18.915978 18336 tablet_service.cc:1467] Processing CreateTablet for tablet 0254a3f73ab14460ba98658fbdf1e885 (DEFAULT_TABLE table=test-table [id=b69be1362a04450b966eb23f2b7f0b4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:36:18.917495 18336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0254a3f73ab14460ba98658fbdf1e885. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:36:18.917507 18467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0254a3f73ab14460ba98658fbdf1e885. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:36:18.919543 18205 tablet_service.cc:1467] Processing CreateTablet for tablet 0254a3f73ab14460ba98658fbdf1e885 (DEFAULT_TABLE table=test-table [id=b69be1362a04450b966eb23f2b7f0b4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:36:18.921028 18205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0254a3f73ab14460ba98658fbdf1e885. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:36:18.931496 18550 tablet_bootstrap.cc:492] T 0254a3f73ab14460ba98658fbdf1e885 P 91b5a5178aac4470b13495d5643597e9: Bootstrap starting.
I20260926 03:36:18.936103 18551 tablet_bootstrap.cc:492] T 0254a3f73ab14460ba98658fbdf1e885 P a33971c3b9044bbcad6977c3d85e7e40: Bootstrap starting.
I20260926 03:36:18.936498 18550 tablet_bootstrap.cc:654] T 0254a3f73ab14460ba98658fbdf1e885 P 91b5a5178aac4470b13495d5643597e9: Neither blocks nor log segments found. Creating new log.
I20260926 03:36:18.937716 18552 tablet_bootstrap.cc:492] T 0254a3f73ab14460ba98658fbdf1e885 P f3d2f65dba1e4a6a91f451b644a806f6: Bootstrap starting.
I20260926 03:36:18.938526 18550 log.cc:824] T 0254a3f73ab14460ba98658fbdf1e885 P 91b5a5178aac4470b13495d5643597e9: Log is configured to *not* fsync() on all Append() calls
I20260926 03:36:18.939970 18551 tablet_bootstrap.cc:654] T 0254a3f73ab14460ba98658fbdf1e885 P a33971c3b9044bbcad6977c3d85e7e40: Neither blocks nor log segments found. Creating new log.
I20260926 03:36:18.941296 18552 tablet_bootstrap.cc:654] T 0254a3f73ab14460ba98658fbdf1e885 P f3d2f65dba1e4a6a91f451b644a806f6: Neither blocks nor log segments found. Creating new log.
I20260926 03:36:18.941493 18551 log.cc:824] T 0254a3f73ab14460ba98658fbdf1e885 P a33971c3b9044bbcad6977c3d85e7e40: Log is configured to *not* fsync() on all Append() calls
I20260926 03:36:18.942183 18550 tablet_bootstrap.cc:492] T 0254a3f73ab14460ba98658fbdf1e885 P 91b5a5178aac4470b13495d5643597e9: No bootstrap required, opened a new log
I20260926 03:36:18.942480 18550 ts_tablet_manager.cc:1397] T 0254a3f73ab14460ba98658fbdf1e885 P 91b5a5178aac4470b13495d5643597e9: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260926 03:36:18.942507 18552 log.cc:824] T 0254a3f73ab14460ba98658fbdf1e885 P f3d2f65dba1e4a6a91f451b644a806f6: Log is configured to *not* fsync() on all Append() calls
I20260926 03:36:18.944645 18551 tablet_bootstrap.cc:492] T 0254a3f73ab14460ba98658fbdf1e885 P a33971c3b9044bbcad6977c3d85e7e40: No bootstrap required, opened a new log
I20260926 03:36:18.944921 18551 ts_tablet_manager.cc:1397] T 0254a3f73ab14460ba98658fbdf1e885 P a33971c3b9044bbcad6977c3d85e7e40: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260926 03:36:18.946838 18552 tablet_bootstrap.cc:492] T 0254a3f73ab14460ba98658fbdf1e885 P f3d2f65dba1e4a6a91f451b644a806f6: No bootstrap required, opened a new log
I20260926 03:36:18.947101 18552 ts_tablet_manager.cc:1397] T 0254a3f73ab14460ba98658fbdf1e885 P f3d2f65dba1e4a6a91f451b644a806f6: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 03:36:18.954555 18552 raft_consensus.cc:348] T 0254a3f73ab14460ba98658fbdf1e885 P f3d2f65dba1e4a6a91f451b644a806f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b5a5178aac4470b13495d5643597e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45503 } } peers { permanent_uuid: "f3d2f65dba1e4a6a91f451b644a806f6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44961 } } peers { permanent_uuid: "a33971c3b9044bbcad6977c3d85e7e40" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38915 } }
I20260926 03:36:18.954566 18550 raft_consensus.cc:348] T 0254a3f73ab14460ba98658fbdf1e885 P 91b5a5178aac4470b13495d5643597e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b5a5178aac4470b13495d5643597e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45503 } } peers { permanent_uuid: "f3d2f65dba1e4a6a91f451b644a806f6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44961 } } peers { permanent_uuid: "a33971c3b9044bbcad6977c3d85e7e40" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38915 } }
I20260926 03:36:18.954952 18552 raft_consensus.cc:374] T 0254a3f73ab14460ba98658fbdf1e885 P f3d2f65dba1e4a6a91f451b644a806f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:36:18.955036 18550 raft_consensus.cc:374] T 0254a3f73ab14460ba98658fbdf1e885 P 91b5a5178aac4470b13495d5643597e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:36:18.955111 18552 raft_consensus.cc:729] T 0254a3f73ab14460ba98658fbdf1e885 P f3d2f65dba1e4a6a91f451b644a806f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3d2f65dba1e4a6a91f451b644a806f6, State: Initialized, Role: FOLLOWER
I20260926 03:36:18.955153 18550 raft_consensus.cc:729] T 0254a3f73ab14460ba98658fbdf1e885 P 91b5a5178aac4470b13495d5643597e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91b5a5178aac4470b13495d5643597e9, State: Initialized, Role: FOLLOWER
I20260926 03:36:18.955775 18552 consensus_queue.cc:260] T 0254a3f73ab14460ba98658fbdf1e885 P f3d2f65dba1e4a6a91f451b644a806f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b5a5178aac4470b13495d5643597e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45503 } } peers { permanent_uuid: "f3d2f65dba1e4a6a91f451b644a806f6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44961 } } peers { permanent_uuid: "a33971c3b9044bbcad6977c3d85e7e40" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38915 } }
I20260926 03:36:18.955789 18550 consensus_queue.cc:260] T 0254a3f73ab14460ba98658fbdf1e885 P 91b5a5178aac4470b13495d5643597e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b5a5178aac4470b13495d5643597e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45503 } } peers { permanent_uuid: "f3d2f65dba1e4a6a91f451b644a806f6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44961 } } peers { permanent_uuid: "a33971c3b9044bbcad6977c3d85e7e40" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38915 } }
I20260926 03:36:18.955742 18551 raft_consensus.cc:348] T 0254a3f73ab14460ba98658fbdf1e885 P a33971c3b9044bbcad6977c3d85e7e40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b5a5178aac4470b13495d5643597e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45503 } } peers { permanent_uuid: "f3d2f65dba1e4a6a91f451b644a806f6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44961 } } peers { permanent_uuid: "a33971c3b9044bbcad6977c3d85e7e40" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38915 } }
I20260926 03:36:18.956163 18551 raft_consensus.cc:374] T 0254a3f73ab14460ba98658fbdf1e885 P a33971c3b9044bbcad6977c3d85e7e40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:36:18.956360 18551 raft_consensus.cc:729] T 0254a3f73ab14460ba98658fbdf1e885 P a33971c3b9044bbcad6977c3d85e7e40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a33971c3b9044bbcad6977c3d85e7e40, State: Initialized, Role: FOLLOWER
I20260926 03:36:18.957211 18551 consensus_queue.cc:260] T 0254a3f73ab14460ba98658fbdf1e885 P a33971c3b9044bbcad6977c3d85e7e40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b5a5178aac4470b13495d5643597e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45503 } } peers { permanent_uuid: "f3d2f65dba1e4a6a91f451b644a806f6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44961 } } peers { permanent_uuid: "a33971c3b9044bbcad6977c3d85e7e40" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38915 } }
I20260926 03:36:18.957525 18552 ts_tablet_manager.cc:1428] T 0254a3f73ab14460ba98658fbdf1e885 P f3d2f65dba1e4a6a91f451b644a806f6: Time spent starting tablet: real 0.010s user 0.007s sys 0.005s
I20260926 03:36:18.962977 18551 ts_tablet_manager.cc:1428] T 0254a3f73ab14460ba98658fbdf1e885 P a33971c3b9044bbcad6977c3d85e7e40: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
I20260926 03:36:18.963963 18532 heartbeater.cc:503] Master 127.0.97.62:38193 was elected leader, sending a full tablet report...
I20260926 03:36:18.963966 18401 heartbeater.cc:503] Master 127.0.97.62:38193 was elected leader, sending a full tablet report...
I20260926 03:36:18.964639 18550 ts_tablet_manager.cc:1428] T 0254a3f73ab14460ba98658fbdf1e885 P 91b5a5178aac4470b13495d5643597e9: Time spent starting tablet: real 0.022s user 0.009s sys 0.010s
I20260926 03:36:18.999799 18558 raft_consensus.cc:482] T 0254a3f73ab14460ba98658fbdf1e885 P a33971c3b9044bbcad6977c3d85e7e40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:36:19.000195 18558 raft_consensus.cc:504] T 0254a3f73ab14460ba98658fbdf1e885 P a33971c3b9044bbcad6977c3d85e7e40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b5a5178aac4470b13495d5643597e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45503 } } peers { permanent_uuid: "f3d2f65dba1e4a6a91f451b644a806f6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44961 } } peers { permanent_uuid: "a33971c3b9044bbcad6977c3d85e7e40" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38915 } }
I20260926 03:36:19.002372 18558 leader_election.cc:288] T 0254a3f73ab14460ba98658fbdf1e885 P a33971c3b9044bbcad6977c3d85e7e40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91b5a5178aac4470b13495d5643597e9 (127.0.97.2:45503), f3d2f65dba1e4a6a91f451b644a806f6 (127.0.97.1:44961)
I20260926 03:36:19.009418 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:19.002870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33827 (local address 127.0.97.2:45503)
0926 03:36:19.003207 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:19.003215 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:19.003240 (+ 25us) server_negotiation.cc:407] Connection header received
0926 03:36:19.003546 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:19.003556 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:19.003679 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:19.003790 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:36:19.004354 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:19.005449 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:19.007451 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:19.008429 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:19.008527 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:19.008671 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:19.008979 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 03:36:19.009115 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260926 03:36:19.009450 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:19.002794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45503 (local address 127.0.97.3:33827)
0926 03:36:19.003126 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:19.003150 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:19.003337 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:19.003840 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:19.003844 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:19.003892 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:36:19.004183 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:19.004196 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:19.005586 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:19.005592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:19.007285 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:19.007305 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:19.008614 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:19.008620 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:19.008803 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:19.008838 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 03:36:19.008939 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 03:36:19.009106 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"spinlock_wait_cycles":26368}
I20260926 03:36:19.009788 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:19.002854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44961 (local address 127.0.97.3:39539)
0926 03:36:19.003792 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:19.003809 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:19.003939 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:19.004341 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:19.004345 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:19.004369 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:36:19.004759 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:19.004772 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:19.006091 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:19.006094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:19.007698 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:19.007712 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:19.009015 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:19.009021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:19.009201 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:19.009238 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 03:36:19.009329 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 03:36:19.009457 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"thread_start_us":572,"threads_started":1}
I20260926 03:36:19.009812 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:19.003100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39539 (local address 127.0.97.1:44961)
0926 03:36:19.003342 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:19.003351 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:19.003930 (+ 579us) server_negotiation.cc:407] Connection header received
0926 03:36:19.004076 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:19.004084 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:19.004190 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:19.004301 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:36:19.004954 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:19.005945 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:19.007900 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:19.008842 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:19.008972 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:19.009080 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:19.009382 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 03:36:19.009530 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 03:36:19.010854 18356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0254a3f73ab14460ba98658fbdf1e885" candidate_uuid: "a33971c3b9044bbcad6977c3d85e7e40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91b5a5178aac4470b13495d5643597e9" is_pre_election: true
I20260926 03:36:19.010936 18225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0254a3f73ab14460ba98658fbdf1e885" candidate_uuid: "a33971c3b9044bbcad6977c3d85e7e40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3d2f65dba1e4a6a91f451b644a806f6" is_pre_election: true
I20260926 03:36:19.011473 18356 raft_consensus.cc:2454] T 0254a3f73ab14460ba98658fbdf1e885 P 91b5a5178aac4470b13495d5643597e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a33971c3b9044bbcad6977c3d85e7e40 in term 0.
I20260926 03:36:19.011507 18225 raft_consensus.cc:2454] T 0254a3f73ab14460ba98658fbdf1e885 P f3d2f65dba1e4a6a91f451b644a806f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a33971c3b9044bbcad6977c3d85e7e40 in term 0.
I20260926 03:36:19.012223 18420 leader_election.cc:302] T 0254a3f73ab14460ba98658fbdf1e885 P a33971c3b9044bbcad6977c3d85e7e40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 91b5a5178aac4470b13495d5643597e9, a33971c3b9044bbcad6977c3d85e7e40, f3d2f65dba1e4a6a91f451b644a806f6; no voters:
I20260926 03:36:19.012710 18558 raft_consensus.cc:2781] T 0254a3f73ab14460ba98658fbdf1e885 P a33971c3b9044bbcad6977c3d85e7e40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:36:19.012862 18558 raft_consensus.cc:482] T 0254a3f73ab14460ba98658fbdf1e885 P a33971c3b9044bbcad6977c3d85e7e40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:36:19.012987 18558 raft_consensus.cc:3037] T 0254a3f73ab14460ba98658fbdf1e885 P a33971c3b9044bbcad6977c3d85e7e40 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:36:19.015435 18558 raft_consensus.cc:504] T 0254a3f73ab14460ba98658fbdf1e885 P a33971c3b9044bbcad6977c3d85e7e40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b5a5178aac4470b13495d5643597e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45503 } } peers { permanent_uuid: "f3d2f65dba1e4a6a91f451b644a806f6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44961 } } peers { permanent_uuid: "a33971c3b9044bbcad6977c3d85e7e40" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38915 } }
I20260926 03:36:19.016193 18558 leader_election.cc:288] T 0254a3f73ab14460ba98658fbdf1e885 P a33971c3b9044bbcad6977c3d85e7e40 [CANDIDATE]: Term 1 election: Requested vote from peers 91b5a5178aac4470b13495d5643597e9 (127.0.97.2:45503), f3d2f65dba1e4a6a91f451b644a806f6 (127.0.97.1:44961)
I20260926 03:36:19.016630 18356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0254a3f73ab14460ba98658fbdf1e885" candidate_uuid: "a33971c3b9044bbcad6977c3d85e7e40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91b5a5178aac4470b13495d5643597e9"
I20260926 03:36:19.016767 18225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0254a3f73ab14460ba98658fbdf1e885" candidate_uuid: "a33971c3b9044bbcad6977c3d85e7e40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3d2f65dba1e4a6a91f451b644a806f6"
I20260926 03:36:19.016844 18356 raft_consensus.cc:3037] T 0254a3f73ab14460ba98658fbdf1e885 P 91b5a5178aac4470b13495d5643597e9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:36:19.016999 18225 raft_consensus.cc:3037] T 0254a3f73ab14460ba98658fbdf1e885 P f3d2f65dba1e4a6a91f451b644a806f6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:36:19.019641 18356 raft_consensus.cc:2454] T 0254a3f73ab14460ba98658fbdf1e885 P 91b5a5178aac4470b13495d5643597e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a33971c3b9044bbcad6977c3d85e7e40 in term 1.
I20260926 03:36:19.019971 18225 raft_consensus.cc:2454] T 0254a3f73ab14460ba98658fbdf1e885 P f3d2f65dba1e4a6a91f451b644a806f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a33971c3b9044bbcad6977c3d85e7e40 in term 1.
I20260926 03:36:19.020128 18423 leader_election.cc:302] T 0254a3f73ab14460ba98658fbdf1e885 P a33971c3b9044bbcad6977c3d85e7e40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91b5a5178aac4470b13495d5643597e9, a33971c3b9044bbcad6977c3d85e7e40; no voters:
I20260926 03:36:19.020453 18558 raft_consensus.cc:2781] T 0254a3f73ab14460ba98658fbdf1e885 P a33971c3b9044bbcad6977c3d85e7e40 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:36:19.020977 18558 raft_consensus.cc:686] T 0254a3f73ab14460ba98658fbdf1e885 P a33971c3b9044bbcad6977c3d85e7e40 [term 1 LEADER]: Becoming Leader. State: Replica: a33971c3b9044bbcad6977c3d85e7e40, State: Running, Role: LEADER
I20260926 03:36:19.021581 18558 consensus_queue.cc:237] T 0254a3f73ab14460ba98658fbdf1e885 P a33971c3b9044bbcad6977c3d85e7e40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b5a5178aac4470b13495d5643597e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45503 } } peers { permanent_uuid: "f3d2f65dba1e4a6a91f451b644a806f6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44961 } } peers { permanent_uuid: "a33971c3b9044bbcad6977c3d85e7e40" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38915 } }
I20260926 03:36:19.026875 18084 catalog_manager.cc:5495] T 0254a3f73ab14460ba98658fbdf1e885 P a33971c3b9044bbcad6977c3d85e7e40 reported cstate change: term changed from 0 to 1, leader changed from <none> to a33971c3b9044bbcad6977c3d85e7e40 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "a33971c3b9044bbcad6977c3d85e7e40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b5a5178aac4470b13495d5643597e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3d2f65dba1e4a6a91f451b644a806f6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a33971c3b9044bbcad6977c3d85e7e40" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38915 } health_report { overall_health: HEALTHY } } }
W20260926 03:36:19.039723 18271 tablet.cc:2369] T 0254a3f73ab14460ba98658fbdf1e885 P f3d2f65dba1e4a6a91f451b644a806f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:36:19.063647 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:19.057810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51586 (local address 127.0.97.3:38915)
0926 03:36:19.058063 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:19.058068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:19.058114 (+ 46us) server_negotiation.cc:407] Connection header received
0926 03:36:19.058287 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:19.058291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:19.058380 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:19.058520 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:36:19.058857 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:19.059740 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:19.061335 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:19.061778 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:19.061876 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:19.062365 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:36:19.062990 (+ 625us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:36:19.063131 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:19.063184 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 03:36:19.063257 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 03:36:19.071440 18402 tablet.cc:2369] T 0254a3f73ab14460ba98658fbdf1e885 P 91b5a5178aac4470b13495d5643597e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:36:19.073318 18356 raft_consensus.cc:1261] T 0254a3f73ab14460ba98658fbdf1e885 P 91b5a5178aac4470b13495d5643597e9 [term 1 FOLLOWER]: Refusing update from remote peer a33971c3b9044bbcad6977c3d85e7e40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:36:19.073328 18225 raft_consensus.cc:1261] T 0254a3f73ab14460ba98658fbdf1e885 P f3d2f65dba1e4a6a91f451b644a806f6 [term 1 FOLLOWER]: Refusing update from remote peer a33971c3b9044bbcad6977c3d85e7e40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:36:19.074359 18558 consensus_queue.cc:1035] T 0254a3f73ab14460ba98658fbdf1e885 P a33971c3b9044bbcad6977c3d85e7e40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3d2f65dba1e4a6a91f451b644a806f6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:36:19.074709 18560 consensus_queue.cc:1035] T 0254a3f73ab14460ba98658fbdf1e885 P a33971c3b9044bbcad6977c3d85e7e40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91b5a5178aac4470b13495d5643597e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 03:36:19.081573 18533 tablet.cc:2369] T 0254a3f73ab14460ba98658fbdf1e885 P a33971c3b9044bbcad6977c3d85e7e40: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:36:19.086333 18563 mvcc.cc:205] Tried to move back new op lower bound from 7333452919072358400 to 7333452918877163520. Current Snapshot: MvccSnapshot[applied={T|T < 7333452919072358400}]
I20260926 03:36:19.099808 18083 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:36:19.100183 18083 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:36:19.104419 18083 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 925b3f50fe904e478d6f1ceecb656177: Sending DeleteTablet for 3 replicas of tablet 0254a3f73ab14460ba98658fbdf1e885
I20260926 03:36:19.106091 18205 tablet_service.cc:1514] Processing DeleteTablet for tablet 0254a3f73ab14460ba98658fbdf1e885 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:36:19 UTC) from {username='slave'} at 127.0.0.1:35124
I20260926 03:36:19.106107 18336 tablet_service.cc:1514] Processing DeleteTablet for tablet 0254a3f73ab14460ba98658fbdf1e885 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:36:19 UTC) from {username='slave'} at 127.0.0.1:34158
I20260926 03:36:19.106981 18467 tablet_service.cc:1514] Processing DeleteTablet for tablet 0254a3f73ab14460ba98658fbdf1e885 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:36:19 UTC) from {username='slave'} at 127.0.0.1:51576
I20260926 03:36:19.108163 18572 tablet_replica.cc:330] T 0254a3f73ab14460ba98658fbdf1e885 P f3d2f65dba1e4a6a91f451b644a806f6: stopping tablet replica
I20260926 03:36:19.108740 18572 raft_consensus.cc:2229] T 0254a3f73ab14460ba98658fbdf1e885 P f3d2f65dba1e4a6a91f451b644a806f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:36:19.109352 18572 raft_consensus.cc:2258] T 0254a3f73ab14460ba98658fbdf1e885 P f3d2f65dba1e4a6a91f451b644a806f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:36:19.109704 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 18143
I20260926 03:36:19.109942 18573 tablet_replica.cc:330] T 0254a3f73ab14460ba98658fbdf1e885 P 91b5a5178aac4470b13495d5643597e9: stopping tablet replica
I20260926 03:36:19.110414 18573 raft_consensus.cc:2229] T 0254a3f73ab14460ba98658fbdf1e885 P 91b5a5178aac4470b13495d5643597e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:36:19.110912 18573 raft_consensus.cc:2258] T 0254a3f73ab14460ba98658fbdf1e885 P 91b5a5178aac4470b13495d5643597e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:36:19.115003 18573 ts_tablet_manager.cc:1905] T 0254a3f73ab14460ba98658fbdf1e885 P 91b5a5178aac4470b13495d5643597e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:36:19.116235 18572 ts_tablet_manager.cc:1905] T 0254a3f73ab14460ba98658fbdf1e885 P f3d2f65dba1e4a6a91f451b644a806f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:36:19.123790 18573 ts_tablet_manager.cc:1918] T 0254a3f73ab14460ba98658fbdf1e885 P 91b5a5178aac4470b13495d5643597e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:36:19.124078 18573 log.cc:1196] T 0254a3f73ab14460ba98658fbdf1e885 P 91b5a5178aac4470b13495d5643597e9: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-1/wal/wals/0254a3f73ab14460ba98658fbdf1e885
I20260926 03:36:19.124792 18573 ts_tablet_manager.cc:1939] T 0254a3f73ab14460ba98658fbdf1e885 P 91b5a5178aac4470b13495d5643597e9: Deleting consensus metadata
I20260926 03:36:19.125108 18574 tablet_replica.cc:330] T 0254a3f73ab14460ba98658fbdf1e885 P a33971c3b9044bbcad6977c3d85e7e40: stopping tablet replica
I20260926 03:36:19.127251 18071 catalog_manager.cc:4841] TS 91b5a5178aac4470b13495d5643597e9 (127.0.97.2:45503): tablet 0254a3f73ab14460ba98658fbdf1e885 (table test-table [id=b69be1362a04450b966eb23f2b7f0b4f]) successfully deleted
I20260926 03:36:19.128027 18574 raft_consensus.cc:2229] T 0254a3f73ab14460ba98658fbdf1e885 P a33971c3b9044bbcad6977c3d85e7e40 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:36:19.129570 18574 raft_consensus.cc:2258] T 0254a3f73ab14460ba98658fbdf1e885 P a33971c3b9044bbcad6977c3d85e7e40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:36:19.130604 18572 ts_tablet_manager.cc:1918] T 0254a3f73ab14460ba98658fbdf1e885 P f3d2f65dba1e4a6a91f451b644a806f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:36:19.130842 18572 log.cc:1196] T 0254a3f73ab14460ba98658fbdf1e885 P f3d2f65dba1e4a6a91f451b644a806f6: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-0/wal/wals/0254a3f73ab14460ba98658fbdf1e885
I20260926 03:36:19.131457 18572 ts_tablet_manager.cc:1939] T 0254a3f73ab14460ba98658fbdf1e885 P f3d2f65dba1e4a6a91f451b644a806f6: Deleting consensus metadata
I20260926 03:36:19.132452 18574 ts_tablet_manager.cc:1905] T 0254a3f73ab14460ba98658fbdf1e885 P a33971c3b9044bbcad6977c3d85e7e40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:36:19.133572 18068 catalog_manager.cc:4841] TS f3d2f65dba1e4a6a91f451b644a806f6 (127.0.97.1:44961): tablet 0254a3f73ab14460ba98658fbdf1e885 (table test-table [id=b69be1362a04450b966eb23f2b7f0b4f]) successfully deleted
I20260926 03:36:19.137058 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:19.111127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35140 (local address 127.0.97.1:44961)
0926 03:36:19.111335 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:19.111346 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:19.128805 (+ 17459us) server_negotiation.cc:407] Connection header received
0926 03:36:19.129047 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:19.129054 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:19.129213 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:19.129319 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:19.130024 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:19.131262 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:19.133306 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:19.133915 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:19.134014 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:19.135422 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:19.135431 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:19.135446 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:36:19.136101 (+ 655us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:19.136152 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:36:19.136174 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:19.136320 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:19.136461 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 03:36:19.136640 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 03:36:19.137845 18265 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:36:19.140767 18574 ts_tablet_manager.cc:1918] T 0254a3f73ab14460ba98658fbdf1e885 P a33971c3b9044bbcad6977c3d85e7e40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:36:19.140945 18574 log.cc:1196] T 0254a3f73ab14460ba98658fbdf1e885 P a33971c3b9044bbcad6977c3d85e7e40: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790393463219689-388-0/minicluster-data/ts-2/wal/wals/0254a3f73ab14460ba98658fbdf1e885
I20260926 03:36:19.141549 18574 ts_tablet_manager.cc:1939] T 0254a3f73ab14460ba98658fbdf1e885 P a33971c3b9044bbcad6977c3d85e7e40: Deleting consensus metadata
I20260926 03:36:19.143597 18071 catalog_manager.cc:4841] TS a33971c3b9044bbcad6977c3d85e7e40 (127.0.97.3:38915): tablet 0254a3f73ab14460ba98658fbdf1e885 (table test-table [id=b69be1362a04450b966eb23f2b7f0b4f]) successfully deleted
I20260926 03:36:19.235560 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 18143
I20260926 03:36:19.256768 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 18274
I20260926 03:36:19.265545 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:19.257702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34170 (local address 127.0.97.2:45503)
0926 03:36:19.257935 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:19.257941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:19.257965 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:36:19.258143 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:19.258152 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:19.258318 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:19.258447 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:19.258860 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:19.260107 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:19.261496 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:19.262185 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:19.262327 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:19.263571 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:19.263586 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:19.263618 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:36:19.264648 (+ 1030us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:19.264732 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:36:19.264761 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:19.264884 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:19.265089 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 03:36:19.265241 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 03:36:19.266793 18396 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:36:19.363390 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 18274
I20260926 03:36:19.390393 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 18405
I20260926 03:36:19.399475 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:19.391314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51602 (local address 127.0.97.3:38915)
0926 03:36:19.391517 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:19.391528 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:19.391572 (+ 44us) server_negotiation.cc:407] Connection header received
0926 03:36:19.391895 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:19.391902 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:19.392021 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:19.392170 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:19.392749 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:19.393591 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:19.395619 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:19.396096 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:19.396185 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:19.397746 (+ 1561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:19.397756 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:19.397774 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:36:19.398533 (+ 759us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:19.398593 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:36:19.398615 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:19.398700 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:19.398912 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 03:36:19.399004 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 03:36:19.400247 18527 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:36:19.537946 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 18405
I20260926 03:36:19.559737 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 18053
I20260926 03:36:19.561138 18114 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:36:19.693176 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 18053
2026-09-26T03:36:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3121 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:36:19 dist-test-slave-dt7p krb5kdc[18582](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:36:19 dist-test-slave-dt7p krb5kdc[18582](info): set up 2 sockets
Sep 26 03:36:19 dist-test-slave-dt7p krb5kdc[18582](info): commencing operation
krb5kdc: starting...
W20260926 03:36:20.821364 388 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:36:21 dist-test-slave-dt7p krb5kdc[18582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:36:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:36:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:36:21.108372 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44709
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44709
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:36:21.491526 18595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:36:21.491832 18595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:36:21.491931 18595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:36:21.499414 18595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:36:21.499536 18595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:36:21.499585 18595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:36:21.499639 18595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:36:21.499686 18595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:36:21.509819 18595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:36:21.511325 18595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:36:21.512984 18595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:36:21.526036 18601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:21.526070 18600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:21.525976 18603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:36:21.527082 18595 server_base.cc:1034] running on GCE node
I20260926 03:36:21.527964 18595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:36:21.529520 18595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:36:21.530751 18595 hybrid_clock.cc:648] HybridClock initialized: now 1790393781530712 us; error 37 us; skew 500 ppm
I20260926 03:36:21.531116 18595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:36:21 dist-test-slave-dt7p krb5kdc[18582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393781, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:36:21.534679 18595 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:36:21.536625 18595 webserver.cc:460] Webserver started at http://127.0.97.62:32915/ using document root <none> and password file <none>
I20260926 03:36:21.537425 18595 fs_manager.cc:359] Metadata directory not provided
I20260926 03:36:21.537539 18595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:36:21.537870 18595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:36:21.541320 18595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "2b6f1e373c9b436bb5d90dac8e8d9c2f"
format_stamp: "Formatted at 2026-09-26 03:36:21 on dist-test-slave-dt7p"
server_key: "41ff84b66889e752a7dc7042c3123a55"
server_key_iv: "c7d8e3372b446d9e53af5292d640335c"
server_key_version: "encryptionkey@0"
I20260926 03:36:21.542151 18595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "2b6f1e373c9b436bb5d90dac8e8d9c2f"
format_stamp: "Formatted at 2026-09-26 03:36:21 on dist-test-slave-dt7p"
server_key: "41ff84b66889e752a7dc7042c3123a55"
server_key_iv: "c7d8e3372b446d9e53af5292d640335c"
server_key_version: "encryptionkey@0"
I20260926 03:36:21.548081 18595 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260926 03:36:21.551833 18610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:21.553323 18595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:36:21.553514 18595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "2b6f1e373c9b436bb5d90dac8e8d9c2f"
format_stamp: "Formatted at 2026-09-26 03:36:21 on dist-test-slave-dt7p"
server_key: "41ff84b66889e752a7dc7042c3123a55"
server_key_iv: "c7d8e3372b446d9e53af5292d640335c"
server_key_version: "encryptionkey@0"
I20260926 03:36:21.553723 18595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:36:21.567003 18595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:36:21.571033 18595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:36:21.571346 18595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:36:21.590641 18595 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44709
I20260926 03:36:21.590662 18661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44709 every 8 connection(s)
I20260926 03:36:21.592175 18595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:36:21.594993 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 18595
I20260926 03:36:21.595244 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:36:21.595799 388 external_mini_cluster.cc:1428] Setting key 6bd5ae9c42a3cd788df65a68e938107f
I20260926 03:36:21.597158 18662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:36:21.610512 18662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b6f1e373c9b436bb5d90dac8e8d9c2f: Bootstrap starting.
Sep 26 03:36:21 dist-test-slave-dt7p krb5kdc[18582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:36:21.614434 18662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b6f1e373c9b436bb5d90dac8e8d9c2f: Neither blocks nor log segments found. Creating new log.
I20260926 03:36:21.615679 18662 log.cc:824] T 00000000000000000000000000000000 P 2b6f1e373c9b436bb5d90dac8e8d9c2f: Log is configured to *not* fsync() on all Append() calls
I20260926 03:36:21.619581 18662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b6f1e373c9b436bb5d90dac8e8d9c2f: No bootstrap required, opened a new log
I20260926 03:36:21.620452 18665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:21.597829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60172 (local address 127.0.97.62:44709)
0926 03:36:21.598731 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:21.598749 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:21.599089 (+ 340us) server_negotiation.cc:407] Connection header received
0926 03:36:21.601086 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:21.601110 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:21.601555 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:21.602014 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:21.603589 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:21.605079 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:21.606524 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:21.607248 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:21.607460 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:21.612293 (+ 4833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:21.612304 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:21.612319 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:36:21.612363 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:21.615716 (+ 3353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:21.616749 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:21.616757 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:21.616766 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:21.617001 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:21.617471 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:21.617475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:21.617477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:21.618038 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:21.618294 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:21.618812 (+ 518us) server_negotiation.cc:299] Negotiation successful
0926 03:36:21.619174 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":391,"threads_started":1}
I20260926 03:36:21.628968 18662 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b6f1e373c9b436bb5d90dac8e8d9c2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b6f1e373c9b436bb5d90dac8e8d9c2f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44709 } }
I20260926 03:36:21.629590 18662 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b6f1e373c9b436bb5d90dac8e8d9c2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:36:21.629709 18662 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b6f1e373c9b436bb5d90dac8e8d9c2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b6f1e373c9b436bb5d90dac8e8d9c2f, State: Initialized, Role: FOLLOWER
I20260926 03:36:21.630363 18662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b6f1e373c9b436bb5d90dac8e8d9c2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b6f1e373c9b436bb5d90dac8e8d9c2f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44709 } }
I20260926 03:36:21.630589 18662 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b6f1e373c9b436bb5d90dac8e8d9c2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:36:21.630697 18662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b6f1e373c9b436bb5d90dac8e8d9c2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:36:21.630888 18662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b6f1e373c9b436bb5d90dac8e8d9c2f [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:36:21.633805 18662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b6f1e373c9b436bb5d90dac8e8d9c2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b6f1e373c9b436bb5d90dac8e8d9c2f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44709 } }
I20260926 03:36:21.634346 18662 leader_election.cc:302] T 00000000000000000000000000000000 P 2b6f1e373c9b436bb5d90dac8e8d9c2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b6f1e373c9b436bb5d90dac8e8d9c2f; no voters:
I20260926 03:36:21.635021 18662 leader_election.cc:288] T 00000000000000000000000000000000 P 2b6f1e373c9b436bb5d90dac8e8d9c2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:36:21.635226 18667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b6f1e373c9b436bb5d90dac8e8d9c2f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:36:21.636118 18667 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b6f1e373c9b436bb5d90dac8e8d9c2f [term 1 LEADER]: Becoming Leader. State: Replica: 2b6f1e373c9b436bb5d90dac8e8d9c2f, State: Running, Role: LEADER
I20260926 03:36:21.636850 18667 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b6f1e373c9b436bb5d90dac8e8d9c2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b6f1e373c9b436bb5d90dac8e8d9c2f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44709 } }
I20260926 03:36:21.637535 18662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b6f1e373c9b436bb5d90dac8e8d9c2f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:36:21.640280 18668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b6f1e373c9b436bb5d90dac8e8d9c2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b6f1e373c9b436bb5d90dac8e8d9c2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b6f1e373c9b436bb5d90dac8e8d9c2f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44709 } } }
I20260926 03:36:21.641144 18668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b6f1e373c9b436bb5d90dac8e8d9c2f [sys.catalog]: This master's current role is: LEADER
I20260926 03:36:21.641470 18669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b6f1e373c9b436bb5d90dac8e8d9c2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b6f1e373c9b436bb5d90dac8e8d9c2f. Latest consensus state: current_term: 1 leader_uuid: "2b6f1e373c9b436bb5d90dac8e8d9c2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b6f1e373c9b436bb5d90dac8e8d9c2f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44709 } } }
I20260926 03:36:21.642071 18669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b6f1e373c9b436bb5d90dac8e8d9c2f [sys.catalog]: This master's current role is: LEADER
I20260926 03:36:21.648264 18675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:36:21.653609 18675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:36:21.663321 18675 catalog_manager.cc:1347] Generated new cluster ID: 8a4dd07bce664d5abe4afd953b733d46
I20260926 03:36:21.663465 18675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:36:21.689339 18675 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:36:21.691534 18675 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:36:21.705605 18675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b6f1e373c9b436bb5d90dac8e8d9c2f: Generated new TSK 0
I20260926 03:36:21.706662 18675 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:36:21.788343 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44709
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:36:22.177249 18688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:36:22.177542 18688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:36:22.177641 18688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:36:22.185019 18688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:36:22.185123 18688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:36:22.185331 18688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:36:22.195163 18688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:36:22.196719 18688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:36:22.198186 18688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:36:22.213065 18696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:22.213440 18693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:22.213193 18694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:36:22.214443 18688 server_base.cc:1034] running on GCE node
I20260926 03:36:22.215122 18688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:36:22.216327 18688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:36:22.217562 18688 hybrid_clock.cc:648] HybridClock initialized: now 1790393782217505 us; error 52 us; skew 500 ppm
I20260926 03:36:22.217929 18688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:36:22 dist-test-slave-dt7p krb5kdc[18582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:36:22.228276 18688 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:36:22.230185 18688 webserver.cc:460] Webserver started at http://127.0.97.1:33731/ using document root <none> and password file <none>
I20260926 03:36:22.231055 18688 fs_manager.cc:359] Metadata directory not provided
I20260926 03:36:22.231216 18688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:36:22.231606 18688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:36:22.235100 18688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-0/data/instance:
uuid: "a064ace4f8a943839dd4cc9156079ec6"
format_stamp: "Formatted at 2026-09-26 03:36:22 on dist-test-slave-dt7p"
server_key: "4a416a69c0143fa8c3ce771836e5bcc1"
server_key_iv: "068014f9704e27da9ebd9a9f8f928da8"
server_key_version: "encryptionkey@0"
I20260926 03:36:22.236061 18688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a064ace4f8a943839dd4cc9156079ec6"
format_stamp: "Formatted at 2026-09-26 03:36:22 on dist-test-slave-dt7p"
server_key: "4a416a69c0143fa8c3ce771836e5bcc1"
server_key_iv: "068014f9704e27da9ebd9a9f8f928da8"
server_key_version: "encryptionkey@0"
I20260926 03:36:22.242121 18688 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260926 03:36:22.246850 18703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:22.248313 18688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:36:22.248510 18688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-0/wal
uuid: "a064ace4f8a943839dd4cc9156079ec6"
format_stamp: "Formatted at 2026-09-26 03:36:22 on dist-test-slave-dt7p"
server_key: "4a416a69c0143fa8c3ce771836e5bcc1"
server_key_iv: "068014f9704e27da9ebd9a9f8f928da8"
server_key_version: "encryptionkey@0"
I20260926 03:36:22.248803 18688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:36:22.281399 18688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:36:22.285977 18688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:36:22.286324 18688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:36:22.287535 18688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:36:22.289139 18688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:36:22.289253 18688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:22.289413 18688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:36:22.289496 18688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:22.326543 18688 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39201
I20260926 03:36:22.326596 18815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39201 every 8 connection(s)
I20260926 03:36:22.328188 18688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:36:22.338140 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 18688
I20260926 03:36:22.338341 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:36:22.338835 388 external_mini_cluster.cc:1428] Setting key 606b4043ea3e1582e9e45d321ccf96eb
Sep 26 03:36:22 dist-test-slave-dt7p krb5kdc[18582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:36:22.351696 18818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:22.331238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46757 (local address 127.0.97.62:44709)
0926 03:36:22.331735 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:22.331741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:22.332857 (+ 1116us) server_negotiation.cc:407] Connection header received
0926 03:36:22.334911 (+ 2054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:22.334918 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:22.335022 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:22.335155 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:22.336908 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:22.337865 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:22.339937 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:22.340572 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:22.340730 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:22.345313 (+ 4583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:22.345319 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:22.345322 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:36:22.345359 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:22.348257 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:22.349127 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:22.349133 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:22.349136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:22.349275 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:22.349740 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:22.349751 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:22.349755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:22.350046 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:22.350225 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:22.351011 (+ 786us) server_negotiation.cc:299] Negotiation successful
0926 03:36:22.351277 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":358,"thread_start_us":222,"threads_started":1}
I20260926 03:36:22.352949 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:22.331862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44709 (local address 127.0.97.1:46757)
0926 03:36:22.332689 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:22.332732 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:22.334603 (+ 1871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:22.335414 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:22.335423 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:22.335997 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:36:22.336731 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:22.336752 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:22.338044 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:22.338048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:22.339754 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:22.339770 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:22.340842 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:22.340852 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:22.341181 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:22.342012 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:36:22.342045 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:36:22.345069 (+ 3024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:36:22.348408 (+ 3339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:36:22.348419 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:36:22.348439 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:36:22.348989 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:36:22.349388 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:36:22.349395 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:36:22.349399 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:36:22.349614 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:36:22.350176 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:36:22.350185 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:36:22.350590 (+ 405us) client_negotiation.cc:769] Sending connection context
0926 03:36:22.350898 (+ 308us) client_negotiation.cc:240] Negotiation successful
0926 03:36:22.351492 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":308,"threads_started":1}
I20260926 03:36:22.354825 18816 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44709
I20260926 03:36:22.355197 18816 heartbeater.cc:464] Registering TS with master...
I20260926 03:36:22.356207 18816 heartbeater.cc:511] Master 127.0.97.62:44709 requested a full tablet report, sending...
I20260926 03:36:22.358839 18627 ts_manager.cc:194] Registered new tserver with Master: a064ace4f8a943839dd4cc9156079ec6 (127.0.97.1:39201)
I20260926 03:36:22.360443 18627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46757
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:36:22.405005 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44709
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:36:22.787320 18822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:36:22.787650 18822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:36:22.787760 18822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:36:22.795497 18822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:36:22.795660 18822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:36:22.795797 18822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:36:22.807125 18822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:36:22.808900 18822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:36:22.810410 18822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:36:22.825137 18827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:22.825307 18830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:22.825469 18828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:36:22.826319 18822 server_base.cc:1034] running on GCE node
I20260926 03:36:22.826973 18822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:36:22.828018 18822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:36:22.829236 18822 hybrid_clock.cc:648] HybridClock initialized: now 1790393782829168 us; error 50 us; skew 500 ppm
I20260926 03:36:22.829597 18822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:36:22 dist-test-slave-dt7p krb5kdc[18582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:36:22.833516 18822 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:36:22.835191 18822 webserver.cc:460] Webserver started at http://127.0.97.2:35471/ using document root <none> and password file <none>
I20260926 03:36:22.836033 18822 fs_manager.cc:359] Metadata directory not provided
I20260926 03:36:22.836207 18822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:36:22.836607 18822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:36:22.840214 18822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-1/data/instance:
uuid: "a739ccaadbcd4e6186b46d39ca33b1a4"
format_stamp: "Formatted at 2026-09-26 03:36:22 on dist-test-slave-dt7p"
server_key: "1104def8a753505f66c4b5c3fbab1c34"
server_key_iv: "fd93dcf557c755338cba94444b696c46"
server_key_version: "encryptionkey@0"
I20260926 03:36:22.841145 18822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a739ccaadbcd4e6186b46d39ca33b1a4"
format_stamp: "Formatted at 2026-09-26 03:36:22 on dist-test-slave-dt7p"
server_key: "1104def8a753505f66c4b5c3fbab1c34"
server_key_iv: "fd93dcf557c755338cba94444b696c46"
server_key_version: "encryptionkey@0"
I20260926 03:36:22.847460 18822 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 03:36:22.851387 18837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:22.853197 18822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:36:22.853401 18822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-1/wal
uuid: "a739ccaadbcd4e6186b46d39ca33b1a4"
format_stamp: "Formatted at 2026-09-26 03:36:22 on dist-test-slave-dt7p"
server_key: "1104def8a753505f66c4b5c3fbab1c34"
server_key_iv: "fd93dcf557c755338cba94444b696c46"
server_key_version: "encryptionkey@0"
I20260926 03:36:22.853673 18822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:36:22.871887 18822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:36:22.877275 18822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:36:22.877549 18822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:36:22.878567 18822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:36:22.880002 18822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:36:22.880075 18822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:22.880190 18822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:36:22.880237 18822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:22.916623 18822 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40659
I20260926 03:36:22.916656 18949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40659 every 8 connection(s)
I20260926 03:36:22.918162 18822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:36:22.923170 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 18822
I20260926 03:36:22.923394 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:36:22.924082 388 external_mini_cluster.cc:1428] Setting key 3b2ef4d28d797a754cee9fe9d181361e
Sep 26 03:36:22 dist-test-slave-dt7p krb5kdc[18582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:36:22.944263 18952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:22.921198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33807 (local address 127.0.97.62:44709)
0926 03:36:22.921716 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:22.921721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:22.923246 (+ 1525us) server_negotiation.cc:407] Connection header received
0926 03:36:22.925440 (+ 2194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:22.925446 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:22.925549 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:22.925683 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:22.927840 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:22.928761 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:22.931371 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:22.931860 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:22.931966 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:22.937032 (+ 5066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:22.937037 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:22.937041 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:36:22.937084 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:22.940756 (+ 3672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:22.941531 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:22.941539 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:22.941545 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:22.941691 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:22.942163 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:22.942191 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:22.942194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:22.942557 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:22.942735 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:22.943540 (+ 805us) server_negotiation.cc:299] Negotiation successful
0926 03:36:22.943812 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":246,"threads_started":1}
I20260926 03:36:22.945582 18953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:22.921802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44709 (local address 127.0.97.2:33807)
0926 03:36:22.922566 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:22.922611 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:22.925129 (+ 2518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:22.925963 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:22.925973 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:22.926695 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:36:22.927628 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:22.927658 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:22.928934 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:22.928942 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:22.931213 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:22.931231 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:22.932122 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:22.932131 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:22.932504 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:22.933533 (+ 1029us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:36:22.933578 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:36:22.936764 (+ 3186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:36:22.940902 (+ 4138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:36:22.940913 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:36:22.940941 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:36:22.941404 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:36:22.941804 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:36:22.941810 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:36:22.941814 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:36:22.942036 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:36:22.942685 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:36:22.942696 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:36:22.943152 (+ 456us) client_negotiation.cc:769] Sending connection context
0926 03:36:22.943446 (+ 294us) client_negotiation.cc:240] Negotiation successful
0926 03:36:22.944087 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":289,"threads_started":1}
I20260926 03:36:22.947394 18950 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44709
I20260926 03:36:22.947805 18950 heartbeater.cc:464] Registering TS with master...
I20260926 03:36:22.948886 18950 heartbeater.cc:511] Master 127.0.97.62:44709 requested a full tablet report, sending...
I20260926 03:36:22.950785 18627 ts_manager.cc:194] Registered new tserver with Master: a739ccaadbcd4e6186b46d39ca33b1a4 (127.0.97.2:40659)
I20260926 03:36:22.952303 18627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33807
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:36:22.994053 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44709
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:36:23.364481 18816 heartbeater.cc:503] Master 127.0.97.62:44709 was elected leader, sending a full tablet report...
W20260926 03:36:23.373811 18956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:36:23.374166 18956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:36:23.374342 18956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:36:23.382457 18956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:36:23.382597 18956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:36:23.382802 18956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:36:23.394263 18956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:36:23.395958 18956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:36:23.397619 18956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:36:23.412714 18961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:23.412653 18962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:23.412668 18964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:36:23.413800 18956 server_base.cc:1034] running on GCE node
I20260926 03:36:23.414537 18956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:36:23.415709 18956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:36:23.416961 18956 hybrid_clock.cc:648] HybridClock initialized: now 1790393783416911 us; error 38 us; skew 500 ppm
I20260926 03:36:23.417377 18956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:36:23 dist-test-slave-dt7p krb5kdc[18582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:36:23.421486 18956 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:36:23.423283 18956 webserver.cc:460] Webserver started at http://127.0.97.3:42281/ using document root <none> and password file <none>
I20260926 03:36:23.424160 18956 fs_manager.cc:359] Metadata directory not provided
I20260926 03:36:23.424324 18956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:36:23.424783 18956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:36:23.428179 18956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-2/data/instance:
uuid: "b8a23b56b7b74a9c9cc4dd1b76af44d8"
format_stamp: "Formatted at 2026-09-26 03:36:23 on dist-test-slave-dt7p"
server_key: "b8dcee9bd1ca51267f98afae536bc7ac"
server_key_iv: "2a7265560869a43e39561d226dc68847"
server_key_version: "encryptionkey@0"
I20260926 03:36:23.429104 18956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b8a23b56b7b74a9c9cc4dd1b76af44d8"
format_stamp: "Formatted at 2026-09-26 03:36:23 on dist-test-slave-dt7p"
server_key: "b8dcee9bd1ca51267f98afae536bc7ac"
server_key_iv: "2a7265560869a43e39561d226dc68847"
server_key_version: "encryptionkey@0"
I20260926 03:36:23.434903 18956 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 03:36:23.439599 18971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:23.441254 18956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:36:23.441459 18956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-2/wal
uuid: "b8a23b56b7b74a9c9cc4dd1b76af44d8"
format_stamp: "Formatted at 2026-09-26 03:36:23 on dist-test-slave-dt7p"
server_key: "b8dcee9bd1ca51267f98afae536bc7ac"
server_key_iv: "2a7265560869a43e39561d226dc68847"
server_key_version: "encryptionkey@0"
I20260926 03:36:23.441708 18956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:36:23.474030 18956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:36:23.478550 18956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:36:23.478888 18956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:36:23.479997 18956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:36:23.481606 18956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:36:23.481714 18956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:23.481866 18956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:36:23.481941 18956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:23.518685 18956 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40851
I20260926 03:36:23.518716 19083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40851 every 8 connection(s)
I20260926 03:36:23.520211 18956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:36:23.524019 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 18956
I20260926 03:36:23.524219 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:36:23.524806 388 external_mini_cluster.cc:1428] Setting key 92f6c4b1fbe07b0c55b285847941ed86
Sep 26 03:36:23 dist-test-slave-dt7p krb5kdc[18582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:36:23.542949 19086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:23.523246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48659 (local address 127.0.97.62:44709)
0926 03:36:23.523869 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:23.523875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:23.524991 (+ 1116us) server_negotiation.cc:407] Connection header received
0926 03:36:23.527303 (+ 2312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:23.527310 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:23.527412 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:23.527537 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:23.529243 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:23.530083 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:23.531712 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:23.532162 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:23.532313 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:23.536193 (+ 3880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:23.536198 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:23.536201 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:36:23.536241 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:23.539535 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:23.540547 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:23.540552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:23.540554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:23.540746 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:23.541184 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:23.541187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:23.541189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:23.541488 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:23.541606 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:23.542430 (+ 824us) server_negotiation.cc:299] Negotiation successful
0926 03:36:23.542635 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":239,"threads_started":1}
I20260926 03:36:23.544200 19087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:23.523866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44709 (local address 127.0.97.3:48659)
0926 03:36:23.524786 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:23.524856 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:23.527018 (+ 2162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:23.527792 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:23.527801 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:23.528336 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:36:23.529066 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:23.529084 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:23.530232 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:23.530238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:23.531552 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:23.531564 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:23.532405 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:23.532412 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:23.532760 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:23.533471 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:36:23.533495 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:36:23.535985 (+ 2490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:36:23.539674 (+ 3689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:36:23.539684 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:36:23.539702 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:36:23.540413 (+ 711us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:36:23.540857 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:36:23.540861 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:36:23.540864 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:36:23.541078 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:36:23.541595 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:36:23.541602 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:36:23.541974 (+ 372us) client_negotiation.cc:769] Sending connection context
0926 03:36:23.542255 (+ 281us) client_negotiation.cc:240] Negotiation successful
0926 03:36:23.542774 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":668,"threads_started":1}
I20260926 03:36:23.545713 19084 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44709
I20260926 03:36:23.546089 19084 heartbeater.cc:464] Registering TS with master...
I20260926 03:36:23.546968 19084 heartbeater.cc:511] Master 127.0.97.62:44709 requested a full tablet report, sending...
I20260926 03:36:23.548552 18627 ts_manager.cc:194] Registered new tserver with Master: b8a23b56b7b74a9c9cc4dd1b76af44d8 (127.0.97.3:40851)
I20260926 03:36:23.549563 18627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48659
I20260926 03:36:23.552719 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:36:23.570763 19086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:23.557432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52630 (local address 127.0.97.62:44709)
0926 03:36:23.557704 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:23.557711 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:23.558122 (+ 411us) server_negotiation.cc:407] Connection header received
0926 03:36:23.558309 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:23.558313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:23.558404 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:23.558487 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:23.560086 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:23.560947 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:23.562277 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:23.562687 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:23.562779 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:23.565467 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:23.565472 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:23.565476 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:36:23.565514 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:23.568255 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:23.568974 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:23.568978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:23.568980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:23.569096 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:23.569435 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:23.569438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:23.569439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:23.569751 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:23.569955 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:23.570209 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 03:36:23.570366 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 03:36:23.574863 18627 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:36:23.582491 18627 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:36:23.622853 19098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:23.597267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40851 (local address 127.0.0.1:48322)
0926 03:36:23.599979 (+ 2712us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:23.600049 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:23.600286 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:23.611382 (+ 11096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:23.611391 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:23.611424 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:36:23.611891 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:23.611905 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:23.615591 (+ 3686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:23.615597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:23.619060 (+ 3463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:23.619074 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:23.621452 (+ 2378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:23.621458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:23.621968 (+ 510us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:23.622031 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 03:36:23.622179 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 03:36:23.622404 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"mutex_wait_us":205,"thread_start_us":23198,"threads_started":1}
I20260926 03:36:23.623760 19099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:23.597659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48322 (local address 127.0.97.3:40851)
0926 03:36:23.610338 (+ 12679us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:23.610347 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:23.610383 (+ 36us) server_negotiation.cc:407] Connection header received
0926 03:36:23.610496 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:23.610504 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:23.610805 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:23.611022 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:36:23.612141 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:23.613931 (+ 1790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:23.619279 (+ 5348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:23.620920 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:23.621047 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:23.621325 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:23.623152 (+ 1827us) server_negotiation.cc:299] Negotiation successful
0926 03:36:23.623367 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12509,"thread_start_us":12329,"threads_started":1}
I20260926 03:36:23.625121 19100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:23.598314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49880 (local address 127.0.97.2:40659)
0926 03:36:23.609565 (+ 11251us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:23.609576 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:23.614784 (+ 5208us) server_negotiation.cc:407] Connection header received
0926 03:36:23.614908 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:23.614918 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:23.615169 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:23.615349 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:36:23.617244 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:23.618886 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:23.621936 (+ 3050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:23.623819 (+ 1883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:23.624065 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:23.624268 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:23.624597 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 03:36:23.624783 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11030,"thread_start_us":11252,"threads_started":1}
I20260926 03:36:23.625141 19101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:23.597613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40659 (local address 127.0.0.1:49880)
0926 03:36:23.614497 (+ 16884us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:23.614517 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:23.614639 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:23.616493 (+ 1854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:23.616498 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:23.616524 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:36:23.617039 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:23.617052 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:23.619054 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:23.619059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:23.621720 (+ 2661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:23.621730 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:23.624186 (+ 2456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:23.624192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:23.624365 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:23.624394 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 03:36:23.624489 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 03:36:23.624648 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16751,"mutex_wait_us":1,"thread_start_us":907,"threads_started":1}
I20260926 03:36:23.628544 19019 tablet_service.cc:1467] Processing CreateTablet for tablet 4e6a99346c0c4e6fbd95aa1f74f975cc (DEFAULT_TABLE table=test-table [id=f2f89d144537425f91cb704d68367de7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:36:23.629045 18885 tablet_service.cc:1467] Processing CreateTablet for tablet 4e6a99346c0c4e6fbd95aa1f74f975cc (DEFAULT_TABLE table=test-table [id=f2f89d144537425f91cb704d68367de7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:36:23.630179 19019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e6a99346c0c4e6fbd95aa1f74f975cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:36:23.630299 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:23.602192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39201 (local address 127.0.0.1:38666)
0926 03:36:23.620783 (+ 18591us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:23.620801 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:23.621003 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:23.621717 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:23.621722 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:23.621747 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:36:23.622242 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:23.622253 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:23.624918 (+ 2665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:23.624924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:23.627143 (+ 2219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:23.627157 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:23.629450 (+ 2293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:23.629456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:23.629630 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:23.629661 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 03:36:23.629754 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 03:36:23.629899 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18445,"mutex_wait_us":1,"thread_start_us":12009,"threads_started":1}
I20260926 03:36:23.630486 18885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e6a99346c0c4e6fbd95aa1f74f975cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:36:23.631112 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:23.603109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38666 (local address 127.0.97.1:39201)
0926 03:36:23.617189 (+ 14080us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:23.617198 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:23.620904 (+ 3706us) server_negotiation.cc:407] Connection header received
0926 03:36:23.621169 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:23.621177 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:23.621421 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:23.621584 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:36:23.622466 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:23.624715 (+ 2249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:23.627352 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:23.628906 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:23.629017 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:23.629264 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:23.630582 (+ 1318us) server_negotiation.cc:299] Negotiation successful
0926 03:36:23.630760 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13861,"thread_start_us":13676,"threads_started":1}
I20260926 03:36:23.634755 18751 tablet_service.cc:1467] Processing CreateTablet for tablet 4e6a99346c0c4e6fbd95aa1f74f975cc (DEFAULT_TABLE table=test-table [id=f2f89d144537425f91cb704d68367de7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:36:23.636548 18751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e6a99346c0c4e6fbd95aa1f74f975cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:36:23.642922 19103 tablet_bootstrap.cc:492] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a739ccaadbcd4e6186b46d39ca33b1a4: Bootstrap starting.
I20260926 03:36:23.646169 19104 tablet_bootstrap.cc:492] T 4e6a99346c0c4e6fbd95aa1f74f975cc P b8a23b56b7b74a9c9cc4dd1b76af44d8: Bootstrap starting.
I20260926 03:36:23.647679 19103 tablet_bootstrap.cc:654] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a739ccaadbcd4e6186b46d39ca33b1a4: Neither blocks nor log segments found. Creating new log.
I20260926 03:36:23.649286 19103 log.cc:824] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a739ccaadbcd4e6186b46d39ca33b1a4: Log is configured to *not* fsync() on all Append() calls
I20260926 03:36:23.650231 19104 tablet_bootstrap.cc:654] T 4e6a99346c0c4e6fbd95aa1f74f975cc P b8a23b56b7b74a9c9cc4dd1b76af44d8: Neither blocks nor log segments found. Creating new log.
I20260926 03:36:23.651502 19104 log.cc:824] T 4e6a99346c0c4e6fbd95aa1f74f975cc P b8a23b56b7b74a9c9cc4dd1b76af44d8: Log is configured to *not* fsync() on all Append() calls
I20260926 03:36:23.652280 19106 tablet_bootstrap.cc:492] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a064ace4f8a943839dd4cc9156079ec6: Bootstrap starting.
I20260926 03:36:23.653847 19103 tablet_bootstrap.cc:492] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a739ccaadbcd4e6186b46d39ca33b1a4: No bootstrap required, opened a new log
I20260926 03:36:23.654312 19103 ts_tablet_manager.cc:1397] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a739ccaadbcd4e6186b46d39ca33b1a4: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20260926 03:36:23.655977 19106 tablet_bootstrap.cc:654] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a064ace4f8a943839dd4cc9156079ec6: Neither blocks nor log segments found. Creating new log.
I20260926 03:36:23.657543 19106 log.cc:824] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a064ace4f8a943839dd4cc9156079ec6: Log is configured to *not* fsync() on all Append() calls
I20260926 03:36:23.659962 19104 tablet_bootstrap.cc:492] T 4e6a99346c0c4e6fbd95aa1f74f975cc P b8a23b56b7b74a9c9cc4dd1b76af44d8: No bootstrap required, opened a new log
I20260926 03:36:23.660318 19104 ts_tablet_manager.cc:1397] T 4e6a99346c0c4e6fbd95aa1f74f975cc P b8a23b56b7b74a9c9cc4dd1b76af44d8: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.007s
I20260926 03:36:23.661229 19106 tablet_bootstrap.cc:492] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a064ace4f8a943839dd4cc9156079ec6: No bootstrap required, opened a new log
I20260926 03:36:23.661504 19106 ts_tablet_manager.cc:1397] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a064ace4f8a943839dd4cc9156079ec6: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260926 03:36:23.664103 19103 raft_consensus.cc:348] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a739ccaadbcd4e6186b46d39ca33b1a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8a23b56b7b74a9c9cc4dd1b76af44d8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40851 } } peers { permanent_uuid: "a739ccaadbcd4e6186b46d39ca33b1a4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40659 } } peers { permanent_uuid: "a064ace4f8a943839dd4cc9156079ec6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39201 } }
I20260926 03:36:23.664552 19103 raft_consensus.cc:374] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a739ccaadbcd4e6186b46d39ca33b1a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:36:23.664681 19103 raft_consensus.cc:729] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a739ccaadbcd4e6186b46d39ca33b1a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a739ccaadbcd4e6186b46d39ca33b1a4, State: Initialized, Role: FOLLOWER
I20260926 03:36:23.665405 19103 consensus_queue.cc:260] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a739ccaadbcd4e6186b46d39ca33b1a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8a23b56b7b74a9c9cc4dd1b76af44d8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40851 } } peers { permanent_uuid: "a739ccaadbcd4e6186b46d39ca33b1a4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40659 } } peers { permanent_uuid: "a064ace4f8a943839dd4cc9156079ec6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39201 } }
I20260926 03:36:23.666669 18950 heartbeater.cc:503] Master 127.0.97.62:44709 was elected leader, sending a full tablet report...
I20260926 03:36:23.667405 19103 ts_tablet_manager.cc:1428] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a739ccaadbcd4e6186b46d39ca33b1a4: Time spent starting tablet: real 0.013s user 0.004s sys 0.008s
W20260926 03:36:23.670336 18951 tablet.cc:2369] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a739ccaadbcd4e6186b46d39ca33b1a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:36:23.671875 19104 raft_consensus.cc:348] T 4e6a99346c0c4e6fbd95aa1f74f975cc P b8a23b56b7b74a9c9cc4dd1b76af44d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8a23b56b7b74a9c9cc4dd1b76af44d8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40851 } } peers { permanent_uuid: "a739ccaadbcd4e6186b46d39ca33b1a4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40659 } } peers { permanent_uuid: "a064ace4f8a943839dd4cc9156079ec6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39201 } }
I20260926 03:36:23.672397 19104 raft_consensus.cc:374] T 4e6a99346c0c4e6fbd95aa1f74f975cc P b8a23b56b7b74a9c9cc4dd1b76af44d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:36:23.672544 19104 raft_consensus.cc:729] T 4e6a99346c0c4e6fbd95aa1f74f975cc P b8a23b56b7b74a9c9cc4dd1b76af44d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8a23b56b7b74a9c9cc4dd1b76af44d8, State: Initialized, Role: FOLLOWER
I20260926 03:36:23.672499 19106 raft_consensus.cc:348] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a064ace4f8a943839dd4cc9156079ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8a23b56b7b74a9c9cc4dd1b76af44d8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40851 } } peers { permanent_uuid: "a739ccaadbcd4e6186b46d39ca33b1a4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40659 } } peers { permanent_uuid: "a064ace4f8a943839dd4cc9156079ec6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39201 } }
I20260926 03:36:23.673113 19106 raft_consensus.cc:374] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a064ace4f8a943839dd4cc9156079ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:36:23.673234 19106 raft_consensus.cc:729] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a064ace4f8a943839dd4cc9156079ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a064ace4f8a943839dd4cc9156079ec6, State: Initialized, Role: FOLLOWER
I20260926 03:36:23.673303 19104 consensus_queue.cc:260] T 4e6a99346c0c4e6fbd95aa1f74f975cc P b8a23b56b7b74a9c9cc4dd1b76af44d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8a23b56b7b74a9c9cc4dd1b76af44d8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40851 } } peers { permanent_uuid: "a739ccaadbcd4e6186b46d39ca33b1a4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40659 } } peers { permanent_uuid: "a064ace4f8a943839dd4cc9156079ec6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39201 } }
I20260926 03:36:23.673913 19106 consensus_queue.cc:260] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a064ace4f8a943839dd4cc9156079ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8a23b56b7b74a9c9cc4dd1b76af44d8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40851 } } peers { permanent_uuid: "a739ccaadbcd4e6186b46d39ca33b1a4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40659 } } peers { permanent_uuid: "a064ace4f8a943839dd4cc9156079ec6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39201 } }
I20260926 03:36:23.674543 19084 heartbeater.cc:503] Master 127.0.97.62:44709 was elected leader, sending a full tablet report...
I20260926 03:36:23.675305 19104 ts_tablet_manager.cc:1428] T 4e6a99346c0c4e6fbd95aa1f74f975cc P b8a23b56b7b74a9c9cc4dd1b76af44d8: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260926 03:36:23.676334 19106 ts_tablet_manager.cc:1428] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a064ace4f8a943839dd4cc9156079ec6: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260926 03:36:23.704941 19109 raft_consensus.cc:482] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a739ccaadbcd4e6186b46d39ca33b1a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:36:23.705368 19109 raft_consensus.cc:504] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a739ccaadbcd4e6186b46d39ca33b1a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8a23b56b7b74a9c9cc4dd1b76af44d8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40851 } } peers { permanent_uuid: "a739ccaadbcd4e6186b46d39ca33b1a4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40659 } } peers { permanent_uuid: "a064ace4f8a943839dd4cc9156079ec6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39201 } }
I20260926 03:36:23.707412 19109 leader_election.cc:288] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a739ccaadbcd4e6186b46d39ca33b1a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8a23b56b7b74a9c9cc4dd1b76af44d8 (127.0.97.3:40851), a064ace4f8a943839dd4cc9156079ec6 (127.0.97.1:39201)
I20260926 03:36:23.714825 19099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:23.707841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44031 (local address 127.0.97.3:40851)
0926 03:36:23.708125 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:23.708134 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:23.709006 (+ 872us) server_negotiation.cc:407] Connection header received
0926 03:36:23.709217 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:23.709221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:23.709293 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:23.709399 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:36:23.710157 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:23.710908 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:23.712884 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:23.713713 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:23.713864 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:23.714000 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:23.714382 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 03:36:23.714528 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260926 03:36:23.714829 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:23.708113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53287 (local address 127.0.97.1:39201)
0926 03:36:23.708397 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:23.708404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:23.708960 (+ 556us) server_negotiation.cc:407] Connection header received
0926 03:36:23.709211 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:23.709216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:23.709286 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:23.709391 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:36:23.710102 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:23.710870 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:23.712877 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:23.713702 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:23.713868 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:23.713995 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:23.714399 (+ 404us) server_negotiation.cc:299] Negotiation successful
0926 03:36:23.714545 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 03:36:23.715225 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:23.707859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40851 (local address 127.0.97.2:44031)
0926 03:36:23.708883 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:23.708897 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:23.709052 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:23.709423 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:23.709427 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:23.709449 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:36:23.709973 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:23.709982 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:23.711059 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:23.711064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:23.712721 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:23.712732 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:23.714000 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:23.714007 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:23.714195 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:23.714235 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 03:36:23.714347 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 03:36:23.714521 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":924,"mutex_wait_us":102,"thread_start_us":515,"threads_started":1}
I20260926 03:36:23.715351 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:23.707772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39201 (local address 127.0.97.2:53287)
0926 03:36:23.708823 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:23.708840 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:23.709039 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:23.709409 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:23.709413 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:23.709437 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:36:23.709924 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:23.709935 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:23.711006 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:23.711014 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:23.712714 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:23.712725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:23.713959 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:23.713966 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:23.714200 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:23.714243 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 03:36:23.714343 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 03:36:23.714521 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"mutex_wait_us":115,"thread_start_us":818,"threads_started":1}
I20260926 03:36:23.716364 18771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e6a99346c0c4e6fbd95aa1f74f975cc" candidate_uuid: "a739ccaadbcd4e6186b46d39ca33b1a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a064ace4f8a943839dd4cc9156079ec6" is_pre_election: true
I20260926 03:36:23.716367 19039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e6a99346c0c4e6fbd95aa1f74f975cc" candidate_uuid: "a739ccaadbcd4e6186b46d39ca33b1a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8a23b56b7b74a9c9cc4dd1b76af44d8" is_pre_election: true
I20260926 03:36:23.717061 19039 raft_consensus.cc:2454] T 4e6a99346c0c4e6fbd95aa1f74f975cc P b8a23b56b7b74a9c9cc4dd1b76af44d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a739ccaadbcd4e6186b46d39ca33b1a4 in term 0.
I20260926 03:36:23.717061 18771 raft_consensus.cc:2454] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a064ace4f8a943839dd4cc9156079ec6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a739ccaadbcd4e6186b46d39ca33b1a4 in term 0.
I20260926 03:36:23.717777 18841 leader_election.cc:302] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a739ccaadbcd4e6186b46d39ca33b1a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a739ccaadbcd4e6186b46d39ca33b1a4, b8a23b56b7b74a9c9cc4dd1b76af44d8; no voters:
I20260926 03:36:23.718237 19109 raft_consensus.cc:2781] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a739ccaadbcd4e6186b46d39ca33b1a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:36:23.718386 19109 raft_consensus.cc:482] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a739ccaadbcd4e6186b46d39ca33b1a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:36:23.718495 19109 raft_consensus.cc:3037] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a739ccaadbcd4e6186b46d39ca33b1a4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:36:23.721222 19109 raft_consensus.cc:504] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a739ccaadbcd4e6186b46d39ca33b1a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8a23b56b7b74a9c9cc4dd1b76af44d8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40851 } } peers { permanent_uuid: "a739ccaadbcd4e6186b46d39ca33b1a4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40659 } } peers { permanent_uuid: "a064ace4f8a943839dd4cc9156079ec6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39201 } }
I20260926 03:36:23.721913 19109 leader_election.cc:288] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a739ccaadbcd4e6186b46d39ca33b1a4 [CANDIDATE]: Term 1 election: Requested vote from peers b8a23b56b7b74a9c9cc4dd1b76af44d8 (127.0.97.3:40851), a064ace4f8a943839dd4cc9156079ec6 (127.0.97.1:39201)
I20260926 03:36:23.722355 19039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e6a99346c0c4e6fbd95aa1f74f975cc" candidate_uuid: "a739ccaadbcd4e6186b46d39ca33b1a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8a23b56b7b74a9c9cc4dd1b76af44d8"
I20260926 03:36:23.722405 18771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e6a99346c0c4e6fbd95aa1f74f975cc" candidate_uuid: "a739ccaadbcd4e6186b46d39ca33b1a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a064ace4f8a943839dd4cc9156079ec6"
I20260926 03:36:23.722610 18771 raft_consensus.cc:3037] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a064ace4f8a943839dd4cc9156079ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:36:23.722620 19039 raft_consensus.cc:3037] T 4e6a99346c0c4e6fbd95aa1f74f975cc P b8a23b56b7b74a9c9cc4dd1b76af44d8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:36:23.726219 18771 raft_consensus.cc:2454] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a064ace4f8a943839dd4cc9156079ec6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a739ccaadbcd4e6186b46d39ca33b1a4 in term 1.
I20260926 03:36:23.726214 19039 raft_consensus.cc:2454] T 4e6a99346c0c4e6fbd95aa1f74f975cc P b8a23b56b7b74a9c9cc4dd1b76af44d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a739ccaadbcd4e6186b46d39ca33b1a4 in term 1.
I20260926 03:36:23.726779 18838 leader_election.cc:302] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a739ccaadbcd4e6186b46d39ca33b1a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a064ace4f8a943839dd4cc9156079ec6, a739ccaadbcd4e6186b46d39ca33b1a4; no voters:
I20260926 03:36:23.727093 19109 raft_consensus.cc:2781] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a739ccaadbcd4e6186b46d39ca33b1a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:36:23.727564 19109 raft_consensus.cc:686] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a739ccaadbcd4e6186b46d39ca33b1a4 [term 1 LEADER]: Becoming Leader. State: Replica: a739ccaadbcd4e6186b46d39ca33b1a4, State: Running, Role: LEADER
I20260926 03:36:23.728101 19109 consensus_queue.cc:237] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a739ccaadbcd4e6186b46d39ca33b1a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8a23b56b7b74a9c9cc4dd1b76af44d8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40851 } } peers { permanent_uuid: "a739ccaadbcd4e6186b46d39ca33b1a4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40659 } } peers { permanent_uuid: "a064ace4f8a943839dd4cc9156079ec6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39201 } }
I20260926 03:36:23.733099 18626 catalog_manager.cc:5495] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a739ccaadbcd4e6186b46d39ca33b1a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a739ccaadbcd4e6186b46d39ca33b1a4 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "a739ccaadbcd4e6186b46d39ca33b1a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8a23b56b7b74a9c9cc4dd1b76af44d8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a739ccaadbcd4e6186b46d39ca33b1a4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a064ace4f8a943839dd4cc9156079ec6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39201 } health_report { overall_health: UNKNOWN } } }
I20260926 03:36:23.764909 19100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:23.759141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49896 (local address 127.0.97.2:40659)
0926 03:36:23.759390 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:23.759395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:23.759413 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:36:23.759514 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:23.759519 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:23.759607 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:23.759753 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:36:23.760172 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:23.761122 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:23.762537 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:23.762959 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:23.763064 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:23.763581 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:36:23.764195 (+ 614us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:36:23.764334 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:23.764443 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 03:36:23.764537 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260926 03:36:23.772125 19085 tablet.cc:2369] T 4e6a99346c0c4e6fbd95aa1f74f975cc P b8a23b56b7b74a9c9cc4dd1b76af44d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:36:23.774356 19039 raft_consensus.cc:1261] T 4e6a99346c0c4e6fbd95aa1f74f975cc P b8a23b56b7b74a9c9cc4dd1b76af44d8 [term 1 FOLLOWER]: Refusing update from remote peer a739ccaadbcd4e6186b46d39ca33b1a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:36:23.774370 18771 raft_consensus.cc:1261] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a064ace4f8a943839dd4cc9156079ec6 [term 1 FOLLOWER]: Refusing update from remote peer a739ccaadbcd4e6186b46d39ca33b1a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:36:23.775501 19114 consensus_queue.cc:1035] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a739ccaadbcd4e6186b46d39ca33b1a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8a23b56b7b74a9c9cc4dd1b76af44d8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:36:23.775889 19109 consensus_queue.cc:1035] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a739ccaadbcd4e6186b46d39ca33b1a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a064ace4f8a943839dd4cc9156079ec6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:36:23.785729 19117 mvcc.cc:205] Tried to move back new op lower bound from 7333452938327605248 to 7333452938154594304. Current Snapshot: MvccSnapshot[applied={T|T < 7333452938327605248}]
I20260926 03:36:23.786871 19118 mvcc.cc:205] Tried to move back new op lower bound from 7333452938327605248 to 7333452938154594304. Current Snapshot: MvccSnapshot[applied={T|T < 7333452938327605248}]
I20260926 03:36:23.802933 18627 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:36:23.803296 18627 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:36:23.807395 18627 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b6f1e373c9b436bb5d90dac8e8d9c2f: Sending DeleteTablet for 3 replicas of tablet 4e6a99346c0c4e6fbd95aa1f74f975cc
I20260926 03:36:23.808825 18751 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e6a99346c0c4e6fbd95aa1f74f975cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:36:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38666
I20260926 03:36:23.808852 19019 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e6a99346c0c4e6fbd95aa1f74f975cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:36:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48322
I20260926 03:36:23.809196 18885 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e6a99346c0c4e6fbd95aa1f74f975cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:36:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49880
I20260926 03:36:23.811254 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 18688
I20260926 03:36:23.821600 19128 tablet_replica.cc:330] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a064ace4f8a943839dd4cc9156079ec6: stopping tablet replica
I20260926 03:36:23.821696 19129 tablet_replica.cc:330] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a739ccaadbcd4e6186b46d39ca33b1a4: stopping tablet replica
I20260926 03:36:23.822156 19129 raft_consensus.cc:2229] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a739ccaadbcd4e6186b46d39ca33b1a4 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:36:23.822157 19128 raft_consensus.cc:2229] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a064ace4f8a943839dd4cc9156079ec6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:36:23.822795 19128 raft_consensus.cc:2258] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a064ace4f8a943839dd4cc9156079ec6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:36:23.822877 19129 raft_consensus.cc:2258] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a739ccaadbcd4e6186b46d39ca33b1a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:36:23.824182 19127 tablet_replica.cc:330] T 4e6a99346c0c4e6fbd95aa1f74f975cc P b8a23b56b7b74a9c9cc4dd1b76af44d8: stopping tablet replica
I20260926 03:36:23.824788 19127 raft_consensus.cc:2229] T 4e6a99346c0c4e6fbd95aa1f74f975cc P b8a23b56b7b74a9c9cc4dd1b76af44d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:36:23.825835 19129 ts_tablet_manager.cc:1905] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a739ccaadbcd4e6186b46d39ca33b1a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:36:23.826735 19128 ts_tablet_manager.cc:1905] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a064ace4f8a943839dd4cc9156079ec6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:36:23.833160 19127 raft_consensus.cc:2258] T 4e6a99346c0c4e6fbd95aa1f74f975cc P b8a23b56b7b74a9c9cc4dd1b76af44d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:36:23.834196 19129 ts_tablet_manager.cc:1918] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a739ccaadbcd4e6186b46d39ca33b1a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:36:23.834367 19129 log.cc:1196] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a739ccaadbcd4e6186b46d39ca33b1a4: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-1/wal/wals/4e6a99346c0c4e6fbd95aa1f74f975cc
I20260926 03:36:23.834969 19129 ts_tablet_manager.cc:1939] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a739ccaadbcd4e6186b46d39ca33b1a4: Deleting consensus metadata
I20260926 03:36:23.836074 19127 ts_tablet_manager.cc:1905] T 4e6a99346c0c4e6fbd95aa1f74f975cc P b8a23b56b7b74a9c9cc4dd1b76af44d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:36:23.836992 18611 catalog_manager.cc:4841] TS a739ccaadbcd4e6186b46d39ca33b1a4 (127.0.97.2:40659): tablet 4e6a99346c0c4e6fbd95aa1f74f975cc (table test-table [id=f2f89d144537425f91cb704d68367de7]) successfully deleted
I20260926 03:36:23.843420 19128 ts_tablet_manager.cc:1918] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a064ace4f8a943839dd4cc9156079ec6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:36:23.843674 19128 log.cc:1196] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a064ace4f8a943839dd4cc9156079ec6: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-0/wal/wals/4e6a99346c0c4e6fbd95aa1f74f975cc
I20260926 03:36:23.844339 19128 ts_tablet_manager.cc:1939] T 4e6a99346c0c4e6fbd95aa1f74f975cc P a064ace4f8a943839dd4cc9156079ec6: Deleting consensus metadata
I20260926 03:36:23.844728 19127 ts_tablet_manager.cc:1918] T 4e6a99346c0c4e6fbd95aa1f74f975cc P b8a23b56b7b74a9c9cc4dd1b76af44d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:36:23.844929 19127 log.cc:1196] T 4e6a99346c0c4e6fbd95aa1f74f975cc P b8a23b56b7b74a9c9cc4dd1b76af44d8: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790393463219689-388-0/minicluster-data/ts-2/wal/wals/4e6a99346c0c4e6fbd95aa1f74f975cc
I20260926 03:36:23.845547 19127 ts_tablet_manager.cc:1939] T 4e6a99346c0c4e6fbd95aa1f74f975cc P b8a23b56b7b74a9c9cc4dd1b76af44d8: Deleting consensus metadata
I20260926 03:36:23.846836 18611 catalog_manager.cc:4841] TS a064ace4f8a943839dd4cc9156079ec6 (127.0.97.1:39201): tablet 4e6a99346c0c4e6fbd95aa1f74f975cc (table test-table [id=f2f89d144537425f91cb704d68367de7]) successfully deleted
I20260926 03:36:23.848241 18614 catalog_manager.cc:4841] TS b8a23b56b7b74a9c9cc4dd1b76af44d8 (127.0.97.3:40851): tablet 4e6a99346c0c4e6fbd95aa1f74f975cc (table test-table [id=f2f89d144537425f91cb704d68367de7]) successfully deleted
Sep 26 03:36:23 dist-test-slave-dt7p krb5kdc[18582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:36:23.858654 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:23.813128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38668 (local address 127.0.97.1:39201)
0926 03:36:23.813407 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:23.813418 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:23.827735 (+ 14317us) server_negotiation.cc:407] Connection header received
0926 03:36:23.828619 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:23.828626 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:23.828845 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:23.835454 (+ 6609us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:23.838524 (+ 3070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:23.839766 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:23.841778 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:23.842493 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:23.842594 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:23.849898 (+ 7304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:23.849914 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:23.849940 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:36:23.849995 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:23.854748 (+ 4753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:23.855734 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:23.855747 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:23.855753 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:23.855955 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:23.856444 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:23.856449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:23.856453 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:23.856992 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:23.857332 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:23.857780 (+ 448us) server_negotiation.cc:299] Negotiation successful
0926 03:36:23.858089 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 03:36:23.860411 18811 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:36:23.976751 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 18688
I20260926 03:36:23.998514 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 18822
Sep 26 03:36:24 dist-test-slave-dt7p krb5kdc[18582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:36:24.017946 19100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:23.999513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49898 (local address 127.0.97.2:40659)
0926 03:36:23.999729 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:23.999734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:23.999760 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:36:24.000020 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:24.000025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:24.000102 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:24.000224 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:24.002111 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:24.003029 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:24.004973 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:24.005418 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:24.005514 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:24.010935 (+ 5421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:24.010953 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:24.010974 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:36:24.011012 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:24.014395 (+ 3383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:24.015123 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:24.015131 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:24.015134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:24.015290 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:24.015710 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:24.015715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:24.015725 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:24.016136 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:24.016535 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:24.017283 (+ 748us) server_negotiation.cc:299] Negotiation successful
0926 03:36:24.017552 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 03:36:24.018779 18945 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:36:24.146654 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 18822
I20260926 03:36:24.168269 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 18956
Sep 26 03:36:24 dist-test-slave-dt7p krb5kdc[18582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:36:24.184361 19099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:24.169198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48330 (local address 127.0.97.3:40851)
0926 03:36:24.169423 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:24.169430 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:24.169453 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:36:24.169698 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:24.169703 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:24.169785 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:24.169895 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:24.171373 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:24.172187 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:24.173677 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:24.174104 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:24.174247 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:24.179086 (+ 4839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:24.179101 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:24.179121 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:36:24.179163 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:24.182005 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:24.182622 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:24.182629 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:24.182632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:24.182761 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:24.183069 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:24.183072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:24.183074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:24.183464 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:24.183700 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:24.183880 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 03:36:24.184051 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:36:24.185105 19079 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:36:24.280593 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 18956
I20260926 03:36:24.299710 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 18595
I20260926 03:36:24.301108 18657 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:36:24.430148 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 18595
2026-09-26T03:36:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4738 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T03:36:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:36:24Z Disabled control of system clock
I20260926 03:36:24.520280 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33225
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 03:36:24.900236 19138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:36:24.900637 19138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:36:24.900795 19138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:36:24.908466 19138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:36:24.908623 19138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:36:24.908711 19138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:36:24.908823 19138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:36:24.908886 19138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:36:24.918462 19138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33225
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:36:24.920203 19138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:36:24.922384 19138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:36:24.935437 19143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:24.935463 19144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:24.935431 19146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:36:24.937019 19138 server_base.cc:1034] running on GCE node
I20260926 03:36:24.937908 19138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:36:24.939499 19138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:36:24.940737 19138 hybrid_clock.cc:648] HybridClock initialized: now 1790393784940660 us; error 60 us; skew 500 ppm
I20260926 03:36:24.941147 19138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:36:24.943733 19138 webserver.cc:460] Webserver started at http://127.0.97.62:37701/ using document root <none> and password file <none>
I20260926 03:36:24.944551 19138 fs_manager.cc:359] Metadata directory not provided
I20260926 03:36:24.944748 19138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:36:24.945127 19138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:36:24.948535 19138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "91d82d06a9804ff0a19fab0b86921773"
format_stamp: "Formatted at 2026-09-26 03:36:24 on dist-test-slave-dt7p"
server_key: "c33f4874a323290286c5a07b92e244f4"
server_key_iv: "cedb550d3047ae2094b96baed1c01402"
server_key_version: "encryptionkey@0"
I20260926 03:36:24.949523 19138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "91d82d06a9804ff0a19fab0b86921773"
format_stamp: "Formatted at 2026-09-26 03:36:24 on dist-test-slave-dt7p"
server_key: "c33f4874a323290286c5a07b92e244f4"
server_key_iv: "cedb550d3047ae2094b96baed1c01402"
server_key_version: "encryptionkey@0"
I20260926 03:36:24.955243 19138 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:36:24.959175 19152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:24.960690 19138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:36:24.960884 19138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "91d82d06a9804ff0a19fab0b86921773"
format_stamp: "Formatted at 2026-09-26 03:36:24 on dist-test-slave-dt7p"
server_key: "c33f4874a323290286c5a07b92e244f4"
server_key_iv: "cedb550d3047ae2094b96baed1c01402"
server_key_version: "encryptionkey@0"
I20260926 03:36:24.961115 19138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:36:24.986604 19138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:36:24.987713 19138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:36:24.988034 19138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:36:25.008936 19138 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33225
I20260926 03:36:25.008978 19203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33225 every 8 connection(s)
I20260926 03:36:25.010510 19138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:36:25.014900 19204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:36:25.018548 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 19138
I20260926 03:36:25.018759 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:36:25.019397 388 external_mini_cluster.cc:1428] Setting key e915625e89090328acef8a51b8c86ede
I20260926 03:36:25.028437 19204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91d82d06a9804ff0a19fab0b86921773: Bootstrap starting.
I20260926 03:36:25.032166 19204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91d82d06a9804ff0a19fab0b86921773: Neither blocks nor log segments found. Creating new log.
I20260926 03:36:25.033501 19204 log.cc:824] T 00000000000000000000000000000000 P 91d82d06a9804ff0a19fab0b86921773: Log is configured to *not* fsync() on all Append() calls
I20260926 03:36:25.034250 19207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:25.021768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40154 (local address 127.0.97.62:33225)
0926 03:36:25.022874 (+ 1106us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:25.022887 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:25.022936 (+ 49us) server_negotiation.cc:407] Connection header received
0926 03:36:25.024864 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:25.024892 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:25.025427 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:25.025860 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:25.026494 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:25.027814 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:25.029378 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:25.030129 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:25.030332 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:25.031603 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:25.031611 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:25.031625 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:36:25.032232 (+ 607us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:25.032366 (+ 134us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:36:25.032397 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:25.032545 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:25.032776 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 03:36:25.033106 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":362,"threads_started":1}
I20260926 03:36:25.036785 19204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91d82d06a9804ff0a19fab0b86921773: No bootstrap required, opened a new log
I20260926 03:36:25.045066 19204 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91d82d06a9804ff0a19fab0b86921773 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91d82d06a9804ff0a19fab0b86921773" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33225 } }
I20260926 03:36:25.045501 19204 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91d82d06a9804ff0a19fab0b86921773 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:36:25.045627 19204 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91d82d06a9804ff0a19fab0b86921773 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d82d06a9804ff0a19fab0b86921773, State: Initialized, Role: FOLLOWER
I20260926 03:36:25.046258 19204 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91d82d06a9804ff0a19fab0b86921773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91d82d06a9804ff0a19fab0b86921773" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33225 } }
I20260926 03:36:25.046517 19204 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91d82d06a9804ff0a19fab0b86921773 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:36:25.046665 19204 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91d82d06a9804ff0a19fab0b86921773 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:36:25.046854 19204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91d82d06a9804ff0a19fab0b86921773 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:36:25.049417 19204 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91d82d06a9804ff0a19fab0b86921773 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91d82d06a9804ff0a19fab0b86921773" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33225 } }
I20260926 03:36:25.049993 19204 leader_election.cc:302] T 00000000000000000000000000000000 P 91d82d06a9804ff0a19fab0b86921773 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91d82d06a9804ff0a19fab0b86921773; no voters:
I20260926 03:36:25.050791 19204 leader_election.cc:288] T 00000000000000000000000000000000 P 91d82d06a9804ff0a19fab0b86921773 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:36:25.050945 19209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91d82d06a9804ff0a19fab0b86921773 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:36:25.051750 19209 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91d82d06a9804ff0a19fab0b86921773 [term 1 LEADER]: Becoming Leader. State: Replica: 91d82d06a9804ff0a19fab0b86921773, State: Running, Role: LEADER
I20260926 03:36:25.052321 19209 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91d82d06a9804ff0a19fab0b86921773 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91d82d06a9804ff0a19fab0b86921773" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33225 } }
I20260926 03:36:25.053234 19204 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91d82d06a9804ff0a19fab0b86921773 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:36:25.055665 19210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91d82d06a9804ff0a19fab0b86921773 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91d82d06a9804ff0a19fab0b86921773" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91d82d06a9804ff0a19fab0b86921773" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33225 } } }
I20260926 03:36:25.056483 19210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91d82d06a9804ff0a19fab0b86921773 [sys.catalog]: This master's current role is: LEADER
I20260926 03:36:25.055693 19211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91d82d06a9804ff0a19fab0b86921773 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91d82d06a9804ff0a19fab0b86921773. Latest consensus state: current_term: 1 leader_uuid: "91d82d06a9804ff0a19fab0b86921773" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91d82d06a9804ff0a19fab0b86921773" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33225 } } }
I20260926 03:36:25.056854 19211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91d82d06a9804ff0a19fab0b86921773 [sys.catalog]: This master's current role is: LEADER
I20260926 03:36:25.058547 19219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:36:25.064592 19219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:36:25.074436 19219 catalog_manager.cc:1347] Generated new cluster ID: 0171e5fbbbc14714b084e6dfb4075829
I20260926 03:36:25.074565 19219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:36:25.102133 19219 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:36:25.104353 19219 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:36:25.114319 19219 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91d82d06a9804ff0a19fab0b86921773: Generated new TSK 0
I20260926 03:36:25.115658 19219 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:36:25.133307 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33225
--builtin_ntp_servers=127.0.97.20:45235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 03:36:25.581411 19228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:36:25.581813 19228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:36:25.582012 19228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:36:25.590458 19228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:36:25.590618 19228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:36:25.590812 19228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:36:25.600620 19228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:36:25.602276 19228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:36:25.604399 19228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:36:25.619354 19233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:25.619647 19236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:25.619400 19234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:36:25.620663 19228 server_base.cc:1034] running on GCE node
I20260926 03:36:25.621428 19228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:36:25.622550 19228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:36:25.623793 19228 hybrid_clock.cc:648] HybridClock initialized: now 1790393785623727 us; error 61 us; skew 500 ppm
I20260926 03:36:25.624151 19228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:36:25.626750 19228 webserver.cc:460] Webserver started at http://127.0.97.1:40153/ using document root <none> and password file <none>
I20260926 03:36:25.627662 19228 fs_manager.cc:359] Metadata directory not provided
I20260926 03:36:25.627803 19228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:36:25.628165 19228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:36:25.631477 19228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-0/data/instance:
uuid: "54972387538845ad8b2e1eff4aa82889"
format_stamp: "Formatted at 2026-09-26 03:36:25 on dist-test-slave-dt7p"
server_key: "c395e5c4058f9534eb8e7bd0b326a34f"
server_key_iv: "2bdd2f672437f9a73238f4f710f38b47"
server_key_version: "encryptionkey@0"
I20260926 03:36:25.632367 19228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-0/wal/instance:
uuid: "54972387538845ad8b2e1eff4aa82889"
format_stamp: "Formatted at 2026-09-26 03:36:25 on dist-test-slave-dt7p"
server_key: "c395e5c4058f9534eb8e7bd0b326a34f"
server_key_iv: "2bdd2f672437f9a73238f4f710f38b47"
server_key_version: "encryptionkey@0"
I20260926 03:36:25.638164 19228 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 03:36:25.641995 19242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:25.643420 19228 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:36:25.643607 19228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-0/wal
uuid: "54972387538845ad8b2e1eff4aa82889"
format_stamp: "Formatted at 2026-09-26 03:36:25 on dist-test-slave-dt7p"
server_key: "c395e5c4058f9534eb8e7bd0b326a34f"
server_key_iv: "2bdd2f672437f9a73238f4f710f38b47"
server_key_version: "encryptionkey@0"
I20260926 03:36:25.643839 19228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:36:25.698390 19228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:36:25.699471 19228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:36:25.699808 19228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:36:25.700969 19228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:36:25.702754 19228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:36:25.702880 19228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:25.703027 19228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:36:25.703121 19228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:25.740428 19228 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34285
I20260926 03:36:25.740495 19354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34285 every 8 connection(s)
I20260926 03:36:25.741981 19228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:36:25.749351 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 19228
I20260926 03:36:25.749545 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:36:25.750140 388 external_mini_cluster.cc:1428] Setting key e9bfcfee2fa5bf1ec1a451fa990c8965
I20260926 03:36:25.753613 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33225
--builtin_ntp_servers=127.0.97.20:45235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 03:36:25.757844 19357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:25.745231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59773 (local address 127.0.97.62:33225)
0926 03:36:25.745744 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:25.745749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:25.747079 (+ 1330us) server_negotiation.cc:407] Connection header received
0926 03:36:25.749274 (+ 2195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:25.749280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:25.749382 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:25.749530 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:25.751077 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:25.752169 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:25.754661 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:25.755100 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:25.755201 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:25.756965 (+ 1764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:25.756970 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:25.756974 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:36:25.756993 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:25.757037 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:36:25.757050 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:25.757122 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:25.757386 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 03:36:25.757513 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":248,"threads_started":1}
I20260926 03:36:25.759310 19358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:25.745913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33225 (local address 127.0.97.1:59773)
0926 03:36:25.746906 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:25.746951 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:25.748951 (+ 2000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:25.749833 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:25.749846 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:25.749987 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:36:25.750885 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:25.750906 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:25.752327 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:25.752334 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:25.754513 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:25.754533 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:25.755340 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:25.755349 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:25.755688 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:25.756527 (+ 839us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:36:25.756533 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:36:25.756619 (+ 86us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:36:25.756743 (+ 124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:36:25.757154 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:36:25.757168 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:36:25.757178 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 03:36:25.757356 (+ 178us) client_negotiation.cc:240] Negotiation successful
0926 03:36:25.757824 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":298,"threads_started":1}
I20260926 03:36:25.761031 19355 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33225
I20260926 03:36:25.761394 19355 heartbeater.cc:464] Registering TS with master...
I20260926 03:36:25.762228 19355 heartbeater.cc:511] Master 127.0.97.62:33225 requested a full tablet report, sending...
I20260926 03:36:25.764623 19169 ts_manager.cc:194] Registered new tserver with Master: 54972387538845ad8b2e1eff4aa82889 (127.0.97.1:34285)
I20260926 03:36:25.766603 19169 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:59773
W20260926 03:36:26.138468 19359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:36:26.138777 19359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:36:26.138868 19359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:36:26.146289 19359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:36:26.146379 19359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:36:26.146486 19359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:36:26.156103 19359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:36:26.157631 19359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:36:26.159699 19359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:36:26.174364 19367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:26.174810 19364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:26.174877 19365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:36:26.175513 19359 server_base.cc:1034] running on GCE node
I20260926 03:36:26.176247 19359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:36:26.177330 19359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:36:26.178565 19359 hybrid_clock.cc:648] HybridClock initialized: now 1790393786178513 us; error 65 us; skew 500 ppm
I20260926 03:36:26.178920 19359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:36:26.181545 19359 webserver.cc:460] Webserver started at http://127.0.97.2:46179/ using document root <none> and password file <none>
I20260926 03:36:26.182333 19359 fs_manager.cc:359] Metadata directory not provided
I20260926 03:36:26.182472 19359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:36:26.182833 19359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:36:26.186064 19359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-1/data/instance:
uuid: "48dbc8b3cfcb4abc96479c9663f762bb"
format_stamp: "Formatted at 2026-09-26 03:36:26 on dist-test-slave-dt7p"
server_key: "f3d1cf209429396d20c93d22954d516e"
server_key_iv: "0d0806e53edcffad56ad2ce2405b76da"
server_key_version: "encryptionkey@0"
I20260926 03:36:26.186941 19359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-1/wal/instance:
uuid: "48dbc8b3cfcb4abc96479c9663f762bb"
format_stamp: "Formatted at 2026-09-26 03:36:26 on dist-test-slave-dt7p"
server_key: "f3d1cf209429396d20c93d22954d516e"
server_key_iv: "0d0806e53edcffad56ad2ce2405b76da"
server_key_version: "encryptionkey@0"
I20260926 03:36:26.192302 19359 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 03:36:26.195959 19373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:26.197458 19359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:36:26.197644 19359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-1/wal
uuid: "48dbc8b3cfcb4abc96479c9663f762bb"
format_stamp: "Formatted at 2026-09-26 03:36:26 on dist-test-slave-dt7p"
server_key: "f3d1cf209429396d20c93d22954d516e"
server_key_iv: "0d0806e53edcffad56ad2ce2405b76da"
server_key_version: "encryptionkey@0"
I20260926 03:36:26.197880 19359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:36:26.220011 19359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:36:26.221074 19359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:36:26.221392 19359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:36:26.222625 19359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:36:26.224381 19359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:36:26.224507 19359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:26.224664 19359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:36:26.224756 19359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:26.261147 19359 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40751
I20260926 03:36:26.261246 19485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40751 every 8 connection(s)
I20260926 03:36:26.262668 19359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:36:26.267912 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 19359
I20260926 03:36:26.268128 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:36:26.268784 388 external_mini_cluster.cc:1428] Setting key d9fbe50abe0313470ae31708bf677b44
I20260926 03:36:26.272716 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33225
--builtin_ntp_servers=127.0.97.20:45235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 03:36:26.279312 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:26.265910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59043 (local address 127.0.97.62:33225)
0926 03:36:26.266470 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:26.266478 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:26.268079 (+ 1601us) server_negotiation.cc:407] Connection header received
0926 03:36:26.270587 (+ 2508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:26.270591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:26.270686 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:26.270803 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:26.272370 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:26.273367 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:26.276064 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:26.276552 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:26.276780 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:26.278377 (+ 1597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:26.278388 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:26.278391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:36:26.278413 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:26.278457 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:36:26.278470 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:26.278573 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:26.278807 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 03:36:26.278918 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":252,"threads_started":1}
I20260926 03:36:26.280814 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:26.266744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33225 (local address 127.0.97.2:59043)
0926 03:36:26.267860 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:26.267918 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:26.270260 (+ 2342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:26.271107 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:26.271124 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:26.271234 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:36:26.272163 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:26.272191 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:26.273540 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:26.273550 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:26.275905 (+ 2355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:26.275922 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:26.276876 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:26.276883 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:26.277148 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:26.277993 (+ 845us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:36:26.277996 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:36:26.278045 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:36:26.278146 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:36:26.278570 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:36:26.278582 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:36:26.278594 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 03:36:26.278795 (+ 201us) client_negotiation.cc:240] Negotiation successful
0926 03:36:26.279264 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":294,"threads_started":1}
I20260926 03:36:26.282538 19486 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33225
I20260926 03:36:26.282894 19486 heartbeater.cc:464] Registering TS with master...
I20260926 03:36:26.283763 19486 heartbeater.cc:511] Master 127.0.97.62:33225 requested a full tablet report, sending...
I20260926 03:36:26.285638 19169 ts_manager.cc:194] Registered new tserver with Master: 48dbc8b3cfcb4abc96479c9663f762bb (127.0.97.2:40751)
I20260926 03:36:26.287024 19169 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:59043
W20260926 03:36:26.653218 19490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:36:26.653617 19490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:36:26.653723 19490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:36:26.662092 19490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:36:26.662207 19490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:36:26.662324 19490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:36:26.672063 19490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:36:26.673720 19490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:36:26.675797 19490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:36:26.690599 19495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:26.690663 19496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:26.690620 19498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:36:26.692327 19490 server_base.cc:1034] running on GCE node
I20260926 03:36:26.693112 19490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:36:26.694267 19490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:36:26.695519 19490 hybrid_clock.cc:648] HybridClock initialized: now 1790393786695452 us; error 66 us; skew 500 ppm
I20260926 03:36:26.695904 19490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:36:26.698592 19490 webserver.cc:460] Webserver started at http://127.0.97.3:33265/ using document root <none> and password file <none>
I20260926 03:36:26.699424 19490 fs_manager.cc:359] Metadata directory not provided
I20260926 03:36:26.699568 19490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:36:26.699954 19490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:36:26.703516 19490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-2/data/instance:
uuid: "d9e1e0cf9a334b98a568cda1a8902147"
format_stamp: "Formatted at 2026-09-26 03:36:26 on dist-test-slave-dt7p"
server_key: "8e8e19e4f517d32369928faeff3b45e1"
server_key_iv: "283b1bcfc9a4084cdd5af396072b9c00"
server_key_version: "encryptionkey@0"
I20260926 03:36:26.704406 19490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d9e1e0cf9a334b98a568cda1a8902147"
format_stamp: "Formatted at 2026-09-26 03:36:26 on dist-test-slave-dt7p"
server_key: "8e8e19e4f517d32369928faeff3b45e1"
server_key_iv: "283b1bcfc9a4084cdd5af396072b9c00"
server_key_version: "encryptionkey@0"
I20260926 03:36:26.710264 19490 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 03:36:26.714309 19504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:26.715873 19490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:36:26.716058 19490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-2/wal
uuid: "d9e1e0cf9a334b98a568cda1a8902147"
format_stamp: "Formatted at 2026-09-26 03:36:26 on dist-test-slave-dt7p"
server_key: "8e8e19e4f517d32369928faeff3b45e1"
server_key_iv: "283b1bcfc9a4084cdd5af396072b9c00"
server_key_version: "encryptionkey@0"
I20260926 03:36:26.716315 19490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:36:26.745033 19490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:36:26.746060 19490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:36:26.746381 19490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:36:26.747572 19490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:36:26.749342 19490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:36:26.749468 19490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:26.749617 19490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:36:26.749709 19490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:26.770680 19355 heartbeater.cc:503] Master 127.0.97.62:33225 was elected leader, sending a full tablet report...
I20260926 03:36:26.788127 19490 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37579
I20260926 03:36:26.788174 19616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37579 every 8 connection(s)
I20260926 03:36:26.789810 19490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:36:26.796383 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 19490
I20260926 03:36:26.796656 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:36:26.797266 388 external_mini_cluster.cc:1428] Setting key a4a433cedf3df90943b8a584d5116fcb
I20260926 03:36:26.804505 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:26.792795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59787 (local address 127.0.97.62:33225)
0926 03:36:26.793332 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:26.793338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:26.794663 (+ 1325us) server_negotiation.cc:407] Connection header received
0926 03:36:26.796736 (+ 2073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:26.796744 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:26.796851 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:26.796993 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:26.798208 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:26.799450 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:26.801332 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:26.801824 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:26.801961 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:26.803504 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:26.803518 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:26.803522 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:36:26.803542 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:26.803588 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:36:26.803601 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:26.803702 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:26.803948 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 03:36:26.804060 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":263,"threads_started":1}
I20260926 03:36:26.805627 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:26.793489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33225 (local address 127.0.97.3:59787)
0926 03:36:26.794475 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:26.794517 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:26.796378 (+ 1861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:26.797213 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:26.797223 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:26.797358 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:36:26.798038 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:26.798056 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:26.799599 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:26.799607 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:26.801167 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:26.801179 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:26.802084 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:26.802091 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:26.802345 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:26.803160 (+ 815us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:36:26.803163 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:36:26.803213 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:36:26.803301 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:36:26.803698 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:36:26.803710 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:36:26.803721 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 03:36:26.803928 (+ 207us) client_negotiation.cc:240] Negotiation successful
0926 03:36:26.804370 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":283,"threads_started":1}
I20260926 03:36:26.807161 19617 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33225
I20260926 03:36:26.807508 19617 heartbeater.cc:464] Registering TS with master...
I20260926 03:36:26.808352 19617 heartbeater.cc:511] Master 127.0.97.62:33225 requested a full tablet report, sending...
I20260926 03:36:26.810027 19169 ts_manager.cc:194] Registered new tserver with Master: d9e1e0cf9a334b98a568cda1a8902147 (127.0.97.3:37579)
I20260926 03:36:26.811017 19169 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:59787
I20260926 03:36:26.812903 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:36:26.824100 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:26.817621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40170 (local address 127.0.97.62:33225)
0926 03:36:26.817868 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:26.817877 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:26.818219 (+ 342us) server_negotiation.cc:407] Connection header received
0926 03:36:26.818385 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:26.818389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:26.818455 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:26.818538 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:26.819032 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:26.819759 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:26.821451 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:26.821955 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:26.822053 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:26.823210 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:26.823215 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:26.823218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:36:26.823237 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:26.823270 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:36:26.823282 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:26.823535 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:26.823680 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 03:36:26.823757 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 03:36:26.828804 19168 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:36:26.836396 19168 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:36:26.878711 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:26.851388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41636 (local address 127.0.97.1:34285)
0926 03:36:26.860982 (+ 9594us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:26.860998 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:26.867874 (+ 6876us) server_negotiation.cc:407] Connection header received
0926 03:36:26.868640 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:26.868649 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:26.868926 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:26.869128 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:36:26.870365 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:26.872249 (+ 1884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:26.875160 (+ 2911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:26.876776 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:26.876925 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:26.877284 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:26.878012 (+ 728us) server_negotiation.cc:299] Negotiation successful
0926 03:36:26.878243 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9341,"thread_start_us":9147,"threads_started":1}
I20260926 03:36:26.878732 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:26.851837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39956 (local address 127.0.97.2:40751)
0926 03:36:26.861551 (+ 9714us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:26.861562 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:26.869087 (+ 7525us) server_negotiation.cc:407] Connection header received
0926 03:36:26.869474 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:26.869485 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:26.869759 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:26.869968 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:36:26.870888 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:26.872463 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:26.875772 (+ 3309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:26.877360 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:26.877504 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:26.877763 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:26.878092 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 03:36:26.878328 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9490,"thread_start_us":9327,"threads_started":1}
I20260926 03:36:26.878979 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:26.850952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51452 (local address 127.0.97.3:37579)
0926 03:36:26.868215 (+ 17263us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:26.868225 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:26.868263 (+ 38us) server_negotiation.cc:407] Connection header received
0926 03:36:26.868407 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:26.868416 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:26.868830 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:26.869670 (+ 840us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:36:26.870310 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:26.872192 (+ 1882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:26.876350 (+ 4158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:26.877597 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:26.877740 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:26.878043 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:26.878282 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 03:36:26.878461 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17046,"thread_start_us":17887,"threads_started":1}
I20260926 03:36:26.879032 19631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:26.850282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37579 (local address 127.0.0.1:51452)
0926 03:36:26.865996 (+ 15714us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:26.866084 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:26.866432 (+ 348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:26.869135 (+ 2703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:26.869143 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:26.869192 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:36:26.870012 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:26.870029 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:26.872352 (+ 2323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:26.872360 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:26.876153 (+ 3793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:26.876165 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:26.877932 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:26.877941 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:26.878101 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:26.878129 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 03:36:26.878215 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 03:36:26.878421 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15219,"mutex_wait_us":405,"spinlock_wait_cycles":384,"thread_start_us":18341,"threads_started":1}
I20260926 03:36:26.879657 19630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:26.851055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40751 (local address 127.0.0.1:39956)
0926 03:36:26.868929 (+ 17874us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:26.868954 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:26.869206 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:26.870098 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:26.870102 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:26.870132 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:36:26.870654 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:26.870668 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:26.872633 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:26.872637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:26.875557 (+ 2920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:26.875571 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:26.877587 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:26.877595 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:26.877775 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:26.877804 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 03:36:26.877910 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 03:36:26.879222 (+ 1312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17735,"mutex_wait_us":250,"spinlock_wait_cycles":10880,"thread_start_us":16871,"threads_started":1}
I20260926 03:36:26.880838 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:26.850591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34285 (local address 127.0.0.1:41636)
0926 03:36:26.867657 (+ 17066us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:26.867685 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:26.868326 (+ 641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:26.869200 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:26.869208 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:26.869238 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:36:26.870012 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:26.870029 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:26.872421 (+ 2392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:26.872426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:26.874897 (+ 2471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:26.874919 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:26.877430 (+ 2511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:26.877439 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:26.877693 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:26.877752 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 03:36:26.879383 (+ 1631us) client_negotiation.cc:240] Negotiation successful
0926 03:36:26.879561 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16868,"mutex_wait_us":243,"thread_start_us":16159,"threads_started":1}
I20260926 03:36:26.884157 19552 tablet_service.cc:1467] Processing CreateTablet for tablet 34065298f1084cf79cf6a5275681f919 (DEFAULT_TABLE table=test-table [id=69792c3d3f1c44239cb8328d2aa1acbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:36:26.884748 19290 tablet_service.cc:1467] Processing CreateTablet for tablet 34065298f1084cf79cf6a5275681f919 (DEFAULT_TABLE table=test-table [id=69792c3d3f1c44239cb8328d2aa1acbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:36:26.884148 19421 tablet_service.cc:1467] Processing CreateTablet for tablet 34065298f1084cf79cf6a5275681f919 (DEFAULT_TABLE table=test-table [id=69792c3d3f1c44239cb8328d2aa1acbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:36:26.886340 19421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34065298f1084cf79cf6a5275681f919. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:36:26.886369 19290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34065298f1084cf79cf6a5275681f919. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:36:26.886360 19552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34065298f1084cf79cf6a5275681f919. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:36:26.902258 19636 tablet_bootstrap.cc:492] T 34065298f1084cf79cf6a5275681f919 P 48dbc8b3cfcb4abc96479c9663f762bb: Bootstrap starting.
I20260926 03:36:26.906073 19636 tablet_bootstrap.cc:654] T 34065298f1084cf79cf6a5275681f919 P 48dbc8b3cfcb4abc96479c9663f762bb: Neither blocks nor log segments found. Creating new log.
I20260926 03:36:26.907219 19636 log.cc:824] T 34065298f1084cf79cf6a5275681f919 P 48dbc8b3cfcb4abc96479c9663f762bb: Log is configured to *not* fsync() on all Append() calls
I20260926 03:36:26.907892 19638 tablet_bootstrap.cc:492] T 34065298f1084cf79cf6a5275681f919 P d9e1e0cf9a334b98a568cda1a8902147: Bootstrap starting.
I20260926 03:36:26.907873 19637 tablet_bootstrap.cc:492] T 34065298f1084cf79cf6a5275681f919 P 54972387538845ad8b2e1eff4aa82889: Bootstrap starting.
I20260926 03:36:26.910964 19636 tablet_bootstrap.cc:492] T 34065298f1084cf79cf6a5275681f919 P 48dbc8b3cfcb4abc96479c9663f762bb: No bootstrap required, opened a new log
I20260926 03:36:26.911342 19636 ts_tablet_manager.cc:1397] T 34065298f1084cf79cf6a5275681f919 P 48dbc8b3cfcb4abc96479c9663f762bb: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 03:36:26.912103 19637 tablet_bootstrap.cc:654] T 34065298f1084cf79cf6a5275681f919 P 54972387538845ad8b2e1eff4aa82889: Neither blocks nor log segments found. Creating new log.
I20260926 03:36:26.912330 19638 tablet_bootstrap.cc:654] T 34065298f1084cf79cf6a5275681f919 P d9e1e0cf9a334b98a568cda1a8902147: Neither blocks nor log segments found. Creating new log.
I20260926 03:36:26.913589 19637 log.cc:824] T 34065298f1084cf79cf6a5275681f919 P 54972387538845ad8b2e1eff4aa82889: Log is configured to *not* fsync() on all Append() calls
I20260926 03:36:26.913589 19638 log.cc:824] T 34065298f1084cf79cf6a5275681f919 P d9e1e0cf9a334b98a568cda1a8902147: Log is configured to *not* fsync() on all Append() calls
I20260926 03:36:26.917390 19638 tablet_bootstrap.cc:492] T 34065298f1084cf79cf6a5275681f919 P d9e1e0cf9a334b98a568cda1a8902147: No bootstrap required, opened a new log
I20260926 03:36:26.917694 19638 ts_tablet_manager.cc:1397] T 34065298f1084cf79cf6a5275681f919 P d9e1e0cf9a334b98a568cda1a8902147: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 03:36:26.917774 19637 tablet_bootstrap.cc:492] T 34065298f1084cf79cf6a5275681f919 P 54972387538845ad8b2e1eff4aa82889: No bootstrap required, opened a new log
I20260926 03:36:26.918062 19637 ts_tablet_manager.cc:1397] T 34065298f1084cf79cf6a5275681f919 P 54972387538845ad8b2e1eff4aa82889: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 03:36:26.920517 19636 raft_consensus.cc:348] T 34065298f1084cf79cf6a5275681f919 P 48dbc8b3cfcb4abc96479c9663f762bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9e1e0cf9a334b98a568cda1a8902147" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37579 } } peers { permanent_uuid: "54972387538845ad8b2e1eff4aa82889" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34285 } } peers { permanent_uuid: "48dbc8b3cfcb4abc96479c9663f762bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40751 } }
I20260926 03:36:26.921139 19636 raft_consensus.cc:374] T 34065298f1084cf79cf6a5275681f919 P 48dbc8b3cfcb4abc96479c9663f762bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:36:26.921250 19636 raft_consensus.cc:729] T 34065298f1084cf79cf6a5275681f919 P 48dbc8b3cfcb4abc96479c9663f762bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48dbc8b3cfcb4abc96479c9663f762bb, State: Initialized, Role: FOLLOWER
I20260926 03:36:26.921991 19636 consensus_queue.cc:260] T 34065298f1084cf79cf6a5275681f919 P 48dbc8b3cfcb4abc96479c9663f762bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9e1e0cf9a334b98a568cda1a8902147" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37579 } } peers { permanent_uuid: "54972387538845ad8b2e1eff4aa82889" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34285 } } peers { permanent_uuid: "48dbc8b3cfcb4abc96479c9663f762bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40751 } }
I20260926 03:36:26.923214 19486 heartbeater.cc:503] Master 127.0.97.62:33225 was elected leader, sending a full tablet report...
I20260926 03:36:26.923992 19636 ts_tablet_manager.cc:1428] T 34065298f1084cf79cf6a5275681f919 P 48dbc8b3cfcb4abc96479c9663f762bb: Time spent starting tablet: real 0.012s user 0.006s sys 0.005s
I20260926 03:36:26.929466 19637 raft_consensus.cc:348] T 34065298f1084cf79cf6a5275681f919 P 54972387538845ad8b2e1eff4aa82889 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9e1e0cf9a334b98a568cda1a8902147" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37579 } } peers { permanent_uuid: "54972387538845ad8b2e1eff4aa82889" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34285 } } peers { permanent_uuid: "48dbc8b3cfcb4abc96479c9663f762bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40751 } }
I20260926 03:36:26.929463 19638 raft_consensus.cc:348] T 34065298f1084cf79cf6a5275681f919 P d9e1e0cf9a334b98a568cda1a8902147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9e1e0cf9a334b98a568cda1a8902147" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37579 } } peers { permanent_uuid: "54972387538845ad8b2e1eff4aa82889" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34285 } } peers { permanent_uuid: "48dbc8b3cfcb4abc96479c9663f762bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40751 } }
I20260926 03:36:26.929968 19638 raft_consensus.cc:374] T 34065298f1084cf79cf6a5275681f919 P d9e1e0cf9a334b98a568cda1a8902147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:36:26.929968 19637 raft_consensus.cc:374] T 34065298f1084cf79cf6a5275681f919 P 54972387538845ad8b2e1eff4aa82889 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:36:26.930081 19637 raft_consensus.cc:729] T 34065298f1084cf79cf6a5275681f919 P 54972387538845ad8b2e1eff4aa82889 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54972387538845ad8b2e1eff4aa82889, State: Initialized, Role: FOLLOWER
I20260926 03:36:26.930088 19638 raft_consensus.cc:729] T 34065298f1084cf79cf6a5275681f919 P d9e1e0cf9a334b98a568cda1a8902147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9e1e0cf9a334b98a568cda1a8902147, State: Initialized, Role: FOLLOWER
I20260926 03:36:26.930784 19637 consensus_queue.cc:260] T 34065298f1084cf79cf6a5275681f919 P 54972387538845ad8b2e1eff4aa82889 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9e1e0cf9a334b98a568cda1a8902147" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37579 } } peers { permanent_uuid: "54972387538845ad8b2e1eff4aa82889" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34285 } } peers { permanent_uuid: "48dbc8b3cfcb4abc96479c9663f762bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40751 } }
I20260926 03:36:26.930783 19638 consensus_queue.cc:260] T 34065298f1084cf79cf6a5275681f919 P d9e1e0cf9a334b98a568cda1a8902147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9e1e0cf9a334b98a568cda1a8902147" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37579 } } peers { permanent_uuid: "54972387538845ad8b2e1eff4aa82889" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34285 } } peers { permanent_uuid: "48dbc8b3cfcb4abc96479c9663f762bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40751 } }
I20260926 03:36:26.932499 19617 heartbeater.cc:503] Master 127.0.97.62:33225 was elected leader, sending a full tablet report...
I20260926 03:36:26.933193 19638 ts_tablet_manager.cc:1428] T 34065298f1084cf79cf6a5275681f919 P d9e1e0cf9a334b98a568cda1a8902147: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260926 03:36:26.934326 19637 ts_tablet_manager.cc:1428] T 34065298f1084cf79cf6a5275681f919 P 54972387538845ad8b2e1eff4aa82889: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260926 03:36:26.937225 19642 raft_consensus.cc:482] T 34065298f1084cf79cf6a5275681f919 P 48dbc8b3cfcb4abc96479c9663f762bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:36:26.937562 19642 raft_consensus.cc:504] T 34065298f1084cf79cf6a5275681f919 P 48dbc8b3cfcb4abc96479c9663f762bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9e1e0cf9a334b98a568cda1a8902147" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37579 } } peers { permanent_uuid: "54972387538845ad8b2e1eff4aa82889" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34285 } } peers { permanent_uuid: "48dbc8b3cfcb4abc96479c9663f762bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40751 } }
I20260926 03:36:26.939339 19642 leader_election.cc:288] T 34065298f1084cf79cf6a5275681f919 P 48dbc8b3cfcb4abc96479c9663f762bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9e1e0cf9a334b98a568cda1a8902147 (127.0.97.3:37579), 54972387538845ad8b2e1eff4aa82889 (127.0.97.1:34285)
I20260926 03:36:26.948083 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:26.940024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43513 (local address 127.0.97.1:34285)
0926 03:36:26.940213 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:26.940219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:26.940552 (+ 333us) server_negotiation.cc:407] Connection header received
0926 03:36:26.940868 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:26.940872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:26.940949 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:26.941027 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:36:26.941848 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:26.942570 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:26.945879 (+ 3309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:26.946974 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:26.947129 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:26.947273 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:26.947633 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 03:36:26.947797 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 03:36:26.948174 19645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:26.939592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34285 (local address 127.0.97.2:43513)
0926 03:36:26.940419 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:26.940449 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:26.940705 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:26.941082 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:26.941090 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:26.941127 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:36:26.941675 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:26.941691 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:26.942722 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:26.942731 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:26.945631 (+ 2900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:26.945648 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:26.947207 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:26.947214 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:26.947438 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:26.947466 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 03:36:26.947582 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 03:36:26.947764 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":496,"threads_started":1}
I20260926 03:36:26.949487 19310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34065298f1084cf79cf6a5275681f919" candidate_uuid: "48dbc8b3cfcb4abc96479c9663f762bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54972387538845ad8b2e1eff4aa82889" is_pre_election: true
I20260926 03:36:26.950217 19310 raft_consensus.cc:2454] T 34065298f1084cf79cf6a5275681f919 P 54972387538845ad8b2e1eff4aa82889 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48dbc8b3cfcb4abc96479c9663f762bb in term 0.
I20260926 03:36:26.951064 19374 leader_election.cc:302] T 34065298f1084cf79cf6a5275681f919 P 48dbc8b3cfcb4abc96479c9663f762bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48dbc8b3cfcb4abc96479c9663f762bb, 54972387538845ad8b2e1eff4aa82889; no voters:
I20260926 03:36:26.951427 19646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:26.939594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37579 (local address 127.0.97.2:52205)
0926 03:36:26.945152 (+ 5558us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:26.945173 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:26.945306 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:26.945637 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:26.945640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:26.945661 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:36:26.946121 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:26.946134 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:26.947215 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:26.947221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:26.949109 (+ 1888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:26.949122 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:26.950516 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:26.950527 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:26.950715 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:26.950749 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 03:36:26.950837 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 03:36:26.950974 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5430,"mutex_wait_us":43,"thread_start_us":5189,"threads_started":1}
I20260926 03:36:26.951603 19642 raft_consensus.cc:2781] T 34065298f1084cf79cf6a5275681f919 P 48dbc8b3cfcb4abc96479c9663f762bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:36:26.951776 19642 raft_consensus.cc:482] T 34065298f1084cf79cf6a5275681f919 P 48dbc8b3cfcb4abc96479c9663f762bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:36:26.951898 19642 raft_consensus.cc:3037] T 34065298f1084cf79cf6a5275681f919 P 48dbc8b3cfcb4abc96479c9663f762bb [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:36:26.952042 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:26.939613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52205 (local address 127.0.97.3:37579)
0926 03:36:26.939862 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:26.939869 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:26.945280 (+ 5411us) server_negotiation.cc:407] Connection header received
0926 03:36:26.945415 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:26.945421 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:26.945511 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:26.945623 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:36:26.946318 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:26.947066 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:26.949307 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:26.950204 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:26.950350 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:26.950504 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:26.950866 (+ 362us) server_negotiation.cc:299] Negotiation successful
0926 03:36:26.951003 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 03:36:26.953023 19572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34065298f1084cf79cf6a5275681f919" candidate_uuid: "48dbc8b3cfcb4abc96479c9663f762bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9e1e0cf9a334b98a568cda1a8902147" is_pre_election: true
I20260926 03:36:26.953578 19572 raft_consensus.cc:2454] T 34065298f1084cf79cf6a5275681f919 P d9e1e0cf9a334b98a568cda1a8902147 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48dbc8b3cfcb4abc96479c9663f762bb in term 0.
I20260926 03:36:26.955032 19642 raft_consensus.cc:504] T 34065298f1084cf79cf6a5275681f919 P 48dbc8b3cfcb4abc96479c9663f762bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9e1e0cf9a334b98a568cda1a8902147" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37579 } } peers { permanent_uuid: "54972387538845ad8b2e1eff4aa82889" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34285 } } peers { permanent_uuid: "48dbc8b3cfcb4abc96479c9663f762bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40751 } }
I20260926 03:36:26.955740 19642 leader_election.cc:288] T 34065298f1084cf79cf6a5275681f919 P 48dbc8b3cfcb4abc96479c9663f762bb [CANDIDATE]: Term 1 election: Requested vote from peers d9e1e0cf9a334b98a568cda1a8902147 (127.0.97.3:37579), 54972387538845ad8b2e1eff4aa82889 (127.0.97.1:34285)
I20260926 03:36:26.956100 19572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34065298f1084cf79cf6a5275681f919" candidate_uuid: "48dbc8b3cfcb4abc96479c9663f762bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9e1e0cf9a334b98a568cda1a8902147"
I20260926 03:36:26.956313 19572 raft_consensus.cc:3037] T 34065298f1084cf79cf6a5275681f919 P d9e1e0cf9a334b98a568cda1a8902147 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:36:26.956357 19310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34065298f1084cf79cf6a5275681f919" candidate_uuid: "48dbc8b3cfcb4abc96479c9663f762bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54972387538845ad8b2e1eff4aa82889"
I20260926 03:36:26.956647 19310 raft_consensus.cc:3037] T 34065298f1084cf79cf6a5275681f919 P 54972387538845ad8b2e1eff4aa82889 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:36:26.959151 19572 raft_consensus.cc:2454] T 34065298f1084cf79cf6a5275681f919 P d9e1e0cf9a334b98a568cda1a8902147 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48dbc8b3cfcb4abc96479c9663f762bb in term 1.
I20260926 03:36:26.959244 19310 raft_consensus.cc:2454] T 34065298f1084cf79cf6a5275681f919 P 54972387538845ad8b2e1eff4aa82889 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48dbc8b3cfcb4abc96479c9663f762bb in term 1.
I20260926 03:36:26.959682 19376 leader_election.cc:302] T 34065298f1084cf79cf6a5275681f919 P 48dbc8b3cfcb4abc96479c9663f762bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48dbc8b3cfcb4abc96479c9663f762bb, d9e1e0cf9a334b98a568cda1a8902147; no voters:
I20260926 03:36:26.960093 19642 raft_consensus.cc:2781] T 34065298f1084cf79cf6a5275681f919 P 48dbc8b3cfcb4abc96479c9663f762bb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:36:26.960664 19642 raft_consensus.cc:686] T 34065298f1084cf79cf6a5275681f919 P 48dbc8b3cfcb4abc96479c9663f762bb [term 1 LEADER]: Becoming Leader. State: Replica: 48dbc8b3cfcb4abc96479c9663f762bb, State: Running, Role: LEADER
I20260926 03:36:26.961308 19642 consensus_queue.cc:237] T 34065298f1084cf79cf6a5275681f919 P 48dbc8b3cfcb4abc96479c9663f762bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9e1e0cf9a334b98a568cda1a8902147" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37579 } } peers { permanent_uuid: "54972387538845ad8b2e1eff4aa82889" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34285 } } peers { permanent_uuid: "48dbc8b3cfcb4abc96479c9663f762bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40751 } }
I20260926 03:36:26.966907 19169 catalog_manager.cc:5495] T 34065298f1084cf79cf6a5275681f919 P 48dbc8b3cfcb4abc96479c9663f762bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 48dbc8b3cfcb4abc96479c9663f762bb (127.0.97.2). New cstate: current_term: 1 leader_uuid: "48dbc8b3cfcb4abc96479c9663f762bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9e1e0cf9a334b98a568cda1a8902147" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54972387538845ad8b2e1eff4aa82889" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48dbc8b3cfcb4abc96479c9663f762bb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40751 } health_report { overall_health: HEALTHY } } }
I20260926 03:36:26.984772 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:26.978626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39958 (local address 127.0.97.2:40751)
0926 03:36:26.978854 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:26.978860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:26.978882 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:36:26.979133 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:26.979138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:26.979249 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:26.979439 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:36:26.979852 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:26.980768 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:26.982241 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:26.982702 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:26.982844 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:26.983459 (+ 615us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:36:26.984085 (+ 626us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:36:26.984290 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:26.984379 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 03:36:26.984429 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:36:26.994325 19572 raft_consensus.cc:1261] T 34065298f1084cf79cf6a5275681f919 P d9e1e0cf9a334b98a568cda1a8902147 [term 1 FOLLOWER]: Refusing update from remote peer 48dbc8b3cfcb4abc96479c9663f762bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:36:26.994326 19310 raft_consensus.cc:1261] T 34065298f1084cf79cf6a5275681f919 P 54972387538845ad8b2e1eff4aa82889 [term 1 FOLLOWER]: Refusing update from remote peer 48dbc8b3cfcb4abc96479c9663f762bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260926 03:36:26.994902 19356 tablet.cc:2369] T 34065298f1084cf79cf6a5275681f919 P 54972387538845ad8b2e1eff4aa82889: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:36:26.995595 19647 consensus_queue.cc:1035] T 34065298f1084cf79cf6a5275681f919 P 48dbc8b3cfcb4abc96479c9663f762bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "54972387538845ad8b2e1eff4aa82889" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:36:26.995992 19642 consensus_queue.cc:1035] T 34065298f1084cf79cf6a5275681f919 P 48dbc8b3cfcb4abc96479c9663f762bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9e1e0cf9a334b98a568cda1a8902147" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:36:27.018044 19652 mvcc.cc:205] Tried to move back new op lower bound from 7333452951517126656 to 7333452951398555648. Current Snapshot: MvccSnapshot[applied={T|T < 7333452951517126656}]
I20260926 03:36:27.027813 19168 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:36:27.028146 19168 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:36:27.032130 19168 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 91d82d06a9804ff0a19fab0b86921773: Sending DeleteTablet for 3 replicas of tablet 34065298f1084cf79cf6a5275681f919
I20260926 03:36:27.033535 19552 tablet_service.cc:1514] Processing DeleteTablet for tablet 34065298f1084cf79cf6a5275681f919 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:36:27 UTC) from {username='slave'} at 127.0.0.1:51452
I20260926 03:36:27.033535 19290 tablet_service.cc:1514] Processing DeleteTablet for tablet 34065298f1084cf79cf6a5275681f919 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:36:27 UTC) from {username='slave'} at 127.0.0.1:41636
I20260926 03:36:27.035531 19421 tablet_service.cc:1514] Processing DeleteTablet for tablet 34065298f1084cf79cf6a5275681f919 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:36:27 UTC) from {username='slave'} at 127.0.0.1:39956
I20260926 03:36:27.037114 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 19228
I20260926 03:36:27.073995 19660 tablet_replica.cc:330] T 34065298f1084cf79cf6a5275681f919 P 54972387538845ad8b2e1eff4aa82889: stopping tablet replica
I20260926 03:36:27.074836 19660 raft_consensus.cc:2229] T 34065298f1084cf79cf6a5275681f919 P 54972387538845ad8b2e1eff4aa82889 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:36:27.075627 19660 raft_consensus.cc:2258] T 34065298f1084cf79cf6a5275681f919 P 54972387538845ad8b2e1eff4aa82889 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:36:27.077440 19662 tablet_replica.cc:330] T 34065298f1084cf79cf6a5275681f919 P 48dbc8b3cfcb4abc96479c9663f762bb: stopping tablet replica
I20260926 03:36:27.077975 19662 raft_consensus.cc:2229] T 34065298f1084cf79cf6a5275681f919 P 48dbc8b3cfcb4abc96479c9663f762bb [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:36:27.078873 19662 raft_consensus.cc:2258] T 34065298f1084cf79cf6a5275681f919 P 48dbc8b3cfcb4abc96479c9663f762bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:36:27.081337 19662 ts_tablet_manager.cc:1905] T 34065298f1084cf79cf6a5275681f919 P 48dbc8b3cfcb4abc96479c9663f762bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:36:27.082129 19660 ts_tablet_manager.cc:1905] T 34065298f1084cf79cf6a5275681f919 P 54972387538845ad8b2e1eff4aa82889: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:36:27.082315 19661 tablet_replica.cc:330] T 34065298f1084cf79cf6a5275681f919 P d9e1e0cf9a334b98a568cda1a8902147: stopping tablet replica
I20260926 03:36:27.082908 19661 raft_consensus.cc:2229] T 34065298f1084cf79cf6a5275681f919 P d9e1e0cf9a334b98a568cda1a8902147 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:36:27.083535 19661 raft_consensus.cc:2258] T 34065298f1084cf79cf6a5275681f919 P d9e1e0cf9a334b98a568cda1a8902147 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:36:27.082275 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:27.039018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41650 (local address 127.0.97.1:34285)
0926 03:36:27.039309 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:27.039319 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:27.066524 (+ 27205us) server_negotiation.cc:407] Connection header received
0926 03:36:27.066932 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:27.066943 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:27.067122 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:27.067230 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:27.068193 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:27.069872 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:27.076985 (+ 7113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:27.077689 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:27.077833 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:27.080268 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:27.080292 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:27.080313 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:36:27.081236 (+ 923us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:27.081330 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:36:27.081361 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:27.081462 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:27.081717 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 03:36:27.081867 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 03:36:27.084697 19350 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:36:27.086843 19661 ts_tablet_manager.cc:1905] T 34065298f1084cf79cf6a5275681f919 P d9e1e0cf9a334b98a568cda1a8902147: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:36:27.090842 19662 ts_tablet_manager.cc:1918] T 34065298f1084cf79cf6a5275681f919 P 48dbc8b3cfcb4abc96479c9663f762bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:36:27.091053 19662 log.cc:1196] T 34065298f1084cf79cf6a5275681f919 P 48dbc8b3cfcb4abc96479c9663f762bb: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-1/wal/wals/34065298f1084cf79cf6a5275681f919
I20260926 03:36:27.091812 19662 ts_tablet_manager.cc:1939] T 34065298f1084cf79cf6a5275681f919 P 48dbc8b3cfcb4abc96479c9663f762bb: Deleting consensus metadata
I20260926 03:36:27.094412 19154 catalog_manager.cc:4841] TS 48dbc8b3cfcb4abc96479c9663f762bb (127.0.97.2:40751): tablet 34065298f1084cf79cf6a5275681f919 (table test-table [id=69792c3d3f1c44239cb8328d2aa1acbd]) successfully deleted
I20260926 03:36:27.095389 19661 ts_tablet_manager.cc:1918] T 34065298f1084cf79cf6a5275681f919 P d9e1e0cf9a334b98a568cda1a8902147: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:36:27.095559 19661 log.cc:1196] T 34065298f1084cf79cf6a5275681f919 P d9e1e0cf9a334b98a568cda1a8902147: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790393463219689-388-0/minicluster-data/ts-2/wal/wals/34065298f1084cf79cf6a5275681f919
I20260926 03:36:27.096119 19661 ts_tablet_manager.cc:1939] T 34065298f1084cf79cf6a5275681f919 P d9e1e0cf9a334b98a568cda1a8902147: Deleting consensus metadata
I20260926 03:36:27.098317 19155 catalog_manager.cc:4841] TS d9e1e0cf9a334b98a568cda1a8902147 (127.0.97.3:37579): tablet 34065298f1084cf79cf6a5275681f919 (table test-table [id=69792c3d3f1c44239cb8328d2aa1acbd]) successfully deleted
I20260926 03:36:27.194380 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 19228
I20260926 03:36:27.215438 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 19359
W20260926 03:36:27.215459 19153 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:34285 (error 108)
I20260926 03:36:27.217998 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:27.217412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34285 (local address 127.0.0.1:41656)
0926 03:36:27.217626 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:27.217768 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 03:36:27.218330 19153 catalog_manager.cc:4584] TS 54972387538845ad8b2e1eff4aa82889 (127.0.97.1:34285): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 34065298f1084cf79cf6a5275681f919: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34285: connect: Connection refused (error 111)
I20260926 03:36:27.224478 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:27.216856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39974 (local address 127.0.97.2:40751)
0926 03:36:27.217112 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:27.217120 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:27.217378 (+ 258us) server_negotiation.cc:407] Connection header received
0926 03:36:27.217652 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:27.217656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:27.217752 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:27.217880 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:27.218358 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:27.219272 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:27.220977 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:27.221434 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:27.221576 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:27.222808 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:27.222818 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:27.222835 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:36:27.223576 (+ 741us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:27.223635 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:36:27.223657 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:27.223794 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:27.223914 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 03:36:27.224030 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 03:36:27.225373 19481 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:36:27.272490 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:27.271830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34285 (local address 127.0.0.1:41658)
0926 03:36:27.272112 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:27.272240 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260926 03:36:27.273731 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:27.273296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34285 (local address 127.0.0.1:41664)
0926 03:36:27.273472 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:27.273562 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 03:36:27.334589 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:27.333913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34285 (local address 127.0.0.1:41676)
0926 03:36:27.334191 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:27.334328 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260926 03:36:27.335902 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:27.335540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34285 (local address 127.0.0.1:41682)
0926 03:36:27.335696 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:27.335773 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 03:36:27.354673 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 19359
I20260926 03:36:27.376554 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 19490
I20260926 03:36:27.388492 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:27.377517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51460 (local address 127.0.97.3:37579)
0926 03:36:27.377714 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:27.377718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:27.377742 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:36:27.377928 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:27.377932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:27.378027 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:27.378121 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:27.378553 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:27.379379 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:27.380984 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:27.381388 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:27.381487 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:27.386890 (+ 5403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:27.386901 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:27.386920 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:36:27.387666 (+ 746us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:27.387730 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:36:27.387754 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:27.387858 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:27.387967 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 03:36:27.388079 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 03:36:27.389319 19612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:36:27.417117 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:27.416428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34285 (local address 127.0.0.1:41694)
0926 03:36:27.416771 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:27.416897 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260926 03:36:27.418437 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:27.417938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34285 (local address 127.0.0.1:41704)
0926 03:36:27.418212 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:27.418295 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
I20260926 03:36:27.485560 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 19490
I20260926 03:36:27.504796 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 19138
I20260926 03:36:27.506146 19199 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:36:27.634049 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:27.633016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34285 (local address 127.0.0.1:41706)
0926 03:36:27.633356 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:27.633485 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260926 03:36:27.634019 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 19138
2026-09-26T03:36:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3203 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:36:27 dist-test-slave-dt7p krb5kdc[19671](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:36:27 dist-test-slave-dt7p krb5kdc[19671](info): set up 2 sockets
Sep 26 03:36:27 dist-test-slave-dt7p krb5kdc[19671](info): commencing operation
krb5kdc: starting...
W20260926 03:36:28.757298 388 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.003s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:36:28 dist-test-slave-dt7p krb5kdc[19671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:36:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:36:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:36:29.048398 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38645
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:36:29.434523 19684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:36:29.434890 19684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:36:29.435066 19684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:36:29.442857 19684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:36:29.442983 19684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:36:29.443104 19684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:36:29.443195 19684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:36:29.443276 19684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:36:29.453967 19684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:36:29.455562 19684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:36:29.457144 19684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:36:29.470368 19690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:29.470448 19689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:29.470474 19692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:36:29.471302 19684 server_base.cc:1034] running on GCE node
I20260926 03:36:29.472188 19684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:36:29.473876 19684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:36:29.475111 19684 hybrid_clock.cc:648] HybridClock initialized: now 1790393789475068 us; error 44 us; skew 500 ppm
I20260926 03:36:29.475490 19684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:36:29 dist-test-slave-dt7p krb5kdc[19671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393789, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:36:29.478992 19684 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:36:29.480806 19684 webserver.cc:460] Webserver started at http://127.0.97.62:34451/ using document root <none> and password file <none>
I20260926 03:36:29.481627 19684 fs_manager.cc:359] Metadata directory not provided
I20260926 03:36:29.481767 19684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:36:29.482101 19684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:36:29.485505 19684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "1b57b5f445be431c894d826860f4c9d7"
format_stamp: "Formatted at 2026-09-26 03:36:29 on dist-test-slave-dt7p"
server_key: "02428506fb790707c40cb7cd7628fd09"
server_key_iv: "4447acd4b0f6425e95d4cac35a87f58d"
server_key_version: "encryptionkey@0"
I20260926 03:36:29.486357 19684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "1b57b5f445be431c894d826860f4c9d7"
format_stamp: "Formatted at 2026-09-26 03:36:29 on dist-test-slave-dt7p"
server_key: "02428506fb790707c40cb7cd7628fd09"
server_key_iv: "4447acd4b0f6425e95d4cac35a87f58d"
server_key_version: "encryptionkey@0"
I20260926 03:36:29.492353 19684 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260926 03:36:29.496433 19699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:29.498040 19684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:36:29.498234 19684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "1b57b5f445be431c894d826860f4c9d7"
format_stamp: "Formatted at 2026-09-26 03:36:29 on dist-test-slave-dt7p"
server_key: "02428506fb790707c40cb7cd7628fd09"
server_key_iv: "4447acd4b0f6425e95d4cac35a87f58d"
server_key_version: "encryptionkey@0"
I20260926 03:36:29.498466 19684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:36:29.513707 19684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:36:29.518204 19684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:36:29.518522 19684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:36:29.539458 19684 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38645
I20260926 03:36:29.539482 19750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38645 every 8 connection(s)
I20260926 03:36:29.541141 19684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:36:29.545655 19751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:36:29.548219 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 19684
I20260926 03:36:29.548442 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:36:29.548977 388 external_mini_cluster.cc:1428] Setting key 2868af2cd1532d2dee269de75c02d723
I20260926 03:36:29.558992 19751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b57b5f445be431c894d826860f4c9d7: Bootstrap starting.
I20260926 03:36:29.563504 19751 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b57b5f445be431c894d826860f4c9d7: Neither blocks nor log segments found. Creating new log.
I20260926 03:36:29.565049 19751 log.cc:824] T 00000000000000000000000000000000 P 1b57b5f445be431c894d826860f4c9d7: Log is configured to *not* fsync() on all Append() calls
Sep 26 03:36:29 dist-test-slave-dt7p krb5kdc[19671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:36:29.568316 19751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b57b5f445be431c894d826860f4c9d7: No bootstrap required, opened a new log
I20260926 03:36:29.574620 19754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:29.551175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59314 (local address 127.0.97.62:38645)
0926 03:36:29.552090 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:29.552103 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:29.552159 (+ 56us) server_negotiation.cc:407] Connection header received
0926 03:36:29.554099 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:29.554126 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:29.554580 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:29.554970 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:29.556976 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:29.558373 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:29.560254 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:29.561034 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:29.561175 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:29.566523 (+ 5348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:29.566532 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:29.566548 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:36:29.566590 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:29.569981 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:29.570792 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:29.570805 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:29.570811 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:29.571003 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:29.571404 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:29.571409 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:29.571421 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:29.572032 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:29.572370 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:29.572909 (+ 539us) server_negotiation.cc:299] Negotiation successful
0926 03:36:29.573418 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":431,"threads_started":1}
I20260926 03:36:29.577600 19751 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b57b5f445be431c894d826860f4c9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b57b5f445be431c894d826860f4c9d7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38645 } }
I20260926 03:36:29.578110 19751 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b57b5f445be431c894d826860f4c9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:36:29.578231 19751 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b57b5f445be431c894d826860f4c9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b57b5f445be431c894d826860f4c9d7, State: Initialized, Role: FOLLOWER
I20260926 03:36:29.578897 19751 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b57b5f445be431c894d826860f4c9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b57b5f445be431c894d826860f4c9d7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38645 } }
I20260926 03:36:29.579146 19751 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b57b5f445be431c894d826860f4c9d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:36:29.579288 19751 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b57b5f445be431c894d826860f4c9d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:36:29.579457 19751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b57b5f445be431c894d826860f4c9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:36:29.582024 19751 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b57b5f445be431c894d826860f4c9d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b57b5f445be431c894d826860f4c9d7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38645 } }
I20260926 03:36:29.582567 19751 leader_election.cc:302] T 00000000000000000000000000000000 P 1b57b5f445be431c894d826860f4c9d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b57b5f445be431c894d826860f4c9d7; no voters:
I20260926 03:36:29.583241 19751 leader_election.cc:288] T 00000000000000000000000000000000 P 1b57b5f445be431c894d826860f4c9d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:36:29.583389 19756 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b57b5f445be431c894d826860f4c9d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:36:29.584699 19756 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b57b5f445be431c894d826860f4c9d7 [term 1 LEADER]: Becoming Leader. State: Replica: 1b57b5f445be431c894d826860f4c9d7, State: Running, Role: LEADER
I20260926 03:36:29.585338 19756 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b57b5f445be431c894d826860f4c9d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b57b5f445be431c894d826860f4c9d7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38645 } }
I20260926 03:36:29.585855 19751 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b57b5f445be431c894d826860f4c9d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:36:29.588358 19757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b57b5f445be431c894d826860f4c9d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b57b5f445be431c894d826860f4c9d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b57b5f445be431c894d826860f4c9d7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38645 } } }
I20260926 03:36:29.588553 19758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b57b5f445be431c894d826860f4c9d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b57b5f445be431c894d826860f4c9d7. Latest consensus state: current_term: 1 leader_uuid: "1b57b5f445be431c894d826860f4c9d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b57b5f445be431c894d826860f4c9d7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38645 } } }
I20260926 03:36:29.588845 19757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b57b5f445be431c894d826860f4c9d7 [sys.catalog]: This master's current role is: LEADER
I20260926 03:36:29.588846 19758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b57b5f445be431c894d826860f4c9d7 [sys.catalog]: This master's current role is: LEADER
I20260926 03:36:29.591565 19765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:36:29.597350 19765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:36:29.609658 19765 catalog_manager.cc:1347] Generated new cluster ID: 84ca384d482d4a9b8e8115475e7f01ba
I20260926 03:36:29.609803 19765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:36:29.622059 19765 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:36:29.623521 19765 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:36:29.632694 19765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b57b5f445be431c894d826860f4c9d7: Generated new TSK 0
I20260926 03:36:29.633690 19765 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:36:29.700366 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38645
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:36:30.090039 19777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:36:30.090361 19777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:36:30.090514 19777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:36:30.098181 19777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:36:30.098311 19777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:36:30.098441 19777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:36:30.109158 19777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:36:30.110800 19777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:36:30.112373 19777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:36:30.130262 19785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:30.130347 19782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:30.130352 19783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:36:30.131592 19777 server_base.cc:1034] running on GCE node
I20260926 03:36:30.132365 19777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:36:30.133613 19777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:36:30.134820 19777 hybrid_clock.cc:648] HybridClock initialized: now 1790393790134747 us; error 58 us; skew 500 ppm
I20260926 03:36:30.135179 19777 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:36:30 dist-test-slave-dt7p krb5kdc[19671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:36:30.138805 19777 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:36:30.140427 19777 webserver.cc:460] Webserver started at http://127.0.97.1:41991/ using document root <none> and password file <none>
I20260926 03:36:30.141273 19777 fs_manager.cc:359] Metadata directory not provided
I20260926 03:36:30.141429 19777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:36:30.141784 19777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:36:30.145169 19777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-0/data/instance:
uuid: "ed01f9ee4431402ea1aaa61b4e99e54b"
format_stamp: "Formatted at 2026-09-26 03:36:30 on dist-test-slave-dt7p"
server_key: "41644f8c854c11ed3a42f477dbd62751"
server_key_iv: "7af14873e17254e95eceb31f466c1d23"
server_key_version: "encryptionkey@0"
I20260926 03:36:30.146059 19777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ed01f9ee4431402ea1aaa61b4e99e54b"
format_stamp: "Formatted at 2026-09-26 03:36:30 on dist-test-slave-dt7p"
server_key: "41644f8c854c11ed3a42f477dbd62751"
server_key_iv: "7af14873e17254e95eceb31f466c1d23"
server_key_version: "encryptionkey@0"
I20260926 03:36:30.151789 19777 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:36:30.155742 19792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:30.157377 19777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:36:30.157562 19777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-0/wal
uuid: "ed01f9ee4431402ea1aaa61b4e99e54b"
format_stamp: "Formatted at 2026-09-26 03:36:30 on dist-test-slave-dt7p"
server_key: "41644f8c854c11ed3a42f477dbd62751"
server_key_iv: "7af14873e17254e95eceb31f466c1d23"
server_key_version: "encryptionkey@0"
I20260926 03:36:30.157778 19777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:36:30.177186 19777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:36:30.182190 19777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:36:30.182535 19777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:36:30.183643 19777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:36:30.185267 19777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:36:30.185380 19777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:30.185535 19777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:36:30.185609 19777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:30.223248 19777 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44139
I20260926 03:36:30.223302 19904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44139 every 8 connection(s)
I20260926 03:36:30.224936 19777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:36:30.230508 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 19777
I20260926 03:36:30.230737 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:36:30.231374 388 external_mini_cluster.cc:1428] Setting key 6b4e65a6af663bc71068de5df1fc0d7b
Sep 26 03:36:30 dist-test-slave-dt7p krb5kdc[19671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:36:30.250300 19907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:30.228120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47811 (local address 127.0.97.62:38645)
0926 03:36:30.228683 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:30.228688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:30.229816 (+ 1128us) server_negotiation.cc:407] Connection header received
0926 03:36:30.232599 (+ 2783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:30.232606 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:30.232730 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:30.232867 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:30.234970 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:30.235823 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:30.238481 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:30.238893 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:30.238992 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:30.243786 (+ 4794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:30.243791 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:30.243795 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:36:30.243833 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:30.246754 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:30.247728 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:30.247732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:30.247735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:30.247860 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:30.248361 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:30.248364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:30.248366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:30.248697 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:30.248821 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:30.249772 (+ 951us) server_negotiation.cc:299] Negotiation successful
0926 03:36:30.249976 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":229,"threads_started":1}
I20260926 03:36:30.251986 19908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:30.228834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38645 (local address 127.0.97.1:47811)
0926 03:36:30.229635 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:30.229677 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:30.232268 (+ 2591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:30.233136 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:30.233145 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:30.233851 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:36:30.234775 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:30.234801 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:30.235980 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:30.235988 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:30.238245 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:30.238263 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:30.239147 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:30.239157 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:30.239517 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:30.240368 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:36:30.240411 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:36:30.243530 (+ 3119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:36:30.246898 (+ 3368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:36:30.246911 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:36:30.246932 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:36:30.247605 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:36:30.247960 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:36:30.247968 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:36:30.247974 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:36:30.248255 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:36:30.248818 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:36:30.248830 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:36:30.249355 (+ 525us) client_negotiation.cc:769] Sending connection context
0926 03:36:30.249687 (+ 332us) client_negotiation.cc:240] Negotiation successful
0926 03:36:30.250347 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":297,"threads_started":1}
I20260926 03:36:30.253762 19905 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38645
I20260926 03:36:30.254129 19905 heartbeater.cc:464] Registering TS with master...
I20260926 03:36:30.255028 19905 heartbeater.cc:511] Master 127.0.97.62:38645 requested a full tablet report, sending...
I20260926 03:36:30.257404 19716 ts_manager.cc:194] Registered new tserver with Master: ed01f9ee4431402ea1aaa61b4e99e54b (127.0.97.1:44139)
I20260926 03:36:30.259061 19716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47811
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:36:30.298267 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38645
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:36:30.673547 19911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:36:30.673892 19911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:36:30.674091 19911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:36:30.682027 19911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:36:30.682188 19911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:36:30.682396 19911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:36:30.693645 19911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:36:30.695324 19911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:36:30.696940 19911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:36:30.711577 19916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:30.711694 19917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:30.711628 19919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:36:30.716331 19911 server_base.cc:1034] running on GCE node
I20260926 03:36:30.717092 19911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:36:30.718179 19911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:36:30.719439 19911 hybrid_clock.cc:648] HybridClock initialized: now 1790393790719384 us; error 58 us; skew 500 ppm
I20260926 03:36:30.719871 19911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:36:30 dist-test-slave-dt7p krb5kdc[19671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:36:30.723485 19911 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:36:30.725222 19911 webserver.cc:460] Webserver started at http://127.0.97.2:35959/ using document root <none> and password file <none>
I20260926 03:36:30.726053 19911 fs_manager.cc:359] Metadata directory not provided
I20260926 03:36:30.726189 19911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:36:30.726539 19911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:36:30.729894 19911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-1/data/instance:
uuid: "698d3939d3d749b48f049d5ccdb157fe"
format_stamp: "Formatted at 2026-09-26 03:36:30 on dist-test-slave-dt7p"
server_key: "6ab7ddb63b44db928e6b5ebb77e22281"
server_key_iv: "97acd231c271fee25fc61c8f39de24c1"
server_key_version: "encryptionkey@0"
I20260926 03:36:30.730777 19911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-1/wal/instance:
uuid: "698d3939d3d749b48f049d5ccdb157fe"
format_stamp: "Formatted at 2026-09-26 03:36:30 on dist-test-slave-dt7p"
server_key: "6ab7ddb63b44db928e6b5ebb77e22281"
server_key_iv: "97acd231c271fee25fc61c8f39de24c1"
server_key_version: "encryptionkey@0"
I20260926 03:36:30.736828 19911 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 03:36:30.740770 19926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:30.742329 19911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:36:30.742507 19911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-1/wal
uuid: "698d3939d3d749b48f049d5ccdb157fe"
format_stamp: "Formatted at 2026-09-26 03:36:30 on dist-test-slave-dt7p"
server_key: "6ab7ddb63b44db928e6b5ebb77e22281"
server_key_iv: "97acd231c271fee25fc61c8f39de24c1"
server_key_version: "encryptionkey@0"
I20260926 03:36:30.742720 19911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:36:30.761169 19911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:36:30.765966 19911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:36:30.766300 19911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:36:30.767410 19911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:36:30.768998 19911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:36:30.769074 19911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:30.769260 19911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:36:30.769330 19911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:30.805523 19911 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44007
I20260926 03:36:30.805562 20038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44007 every 8 connection(s)
I20260926 03:36:30.807004 19911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:36:30.816805 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 19911
I20260926 03:36:30.817014 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:36:30.817489 388 external_mini_cluster.cc:1428] Setting key 409df79c116ef1b8a44174915dc808ab
Sep 26 03:36:30 dist-test-slave-dt7p krb5kdc[19671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:36:30.830626 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:30.809928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55509 (local address 127.0.97.62:38645)
0926 03:36:30.810491 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:30.810499 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:30.811546 (+ 1047us) server_negotiation.cc:407] Connection header received
0926 03:36:30.813819 (+ 2273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:30.813824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:30.813908 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:30.814014 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:30.815609 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:30.816505 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:30.818394 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:30.818955 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:30.819058 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:30.823401 (+ 4343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:30.823409 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:30.823417 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:36:30.823468 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:30.827219 (+ 3751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:30.827941 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:30.827950 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:30.827956 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:30.828143 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:30.828541 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:30.828548 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:30.828554 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:30.828975 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:30.829133 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:30.829793 (+ 660us) server_negotiation.cc:299] Negotiation successful
0926 03:36:30.830080 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":257,"threads_started":1}
I20260926 03:36:30.831355 20042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:30.810520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38645 (local address 127.0.97.2:55509)
0926 03:36:30.811376 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:30.811420 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:30.813521 (+ 2101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:30.814258 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:30.814272 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:30.814826 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:36:30.815456 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:30.815473 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:30.816674 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:30.816678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:30.818235 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:30.818249 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:30.819214 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:30.819222 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:30.819508 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:30.820407 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:36:30.820432 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:36:30.823147 (+ 2715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:36:30.827369 (+ 4222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:36:30.827375 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:36:30.827389 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:36:30.827820 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:36:30.828256 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:36:30.828259 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:36:30.828261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:36:30.828438 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:36:30.829086 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:36:30.829092 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:36:30.829418 (+ 326us) client_negotiation.cc:769] Sending connection context
0926 03:36:30.829672 (+ 254us) client_negotiation.cc:240] Negotiation successful
0926 03:36:30.830072 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":333,"threads_started":1}
I20260926 03:36:30.833101 20039 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38645
I20260926 03:36:30.833454 20039 heartbeater.cc:464] Registering TS with master...
I20260926 03:36:30.834265 20039 heartbeater.cc:511] Master 127.0.97.62:38645 requested a full tablet report, sending...
I20260926 03:36:30.835896 19716 ts_manager.cc:194] Registered new tserver with Master: 698d3939d3d749b48f049d5ccdb157fe (127.0.97.2:44007)
I20260926 03:36:30.836858 19716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55509
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:36:30.883253 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38645
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:36:31.253955 20045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:36:31.254288 20045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:36:31.254454 20045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:36:31.262033 20045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:36:31.262182 20045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:36:31.262375 20045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:36:31.262483 19905 heartbeater.cc:503] Master 127.0.97.62:38645 was elected leader, sending a full tablet report...
I20260926 03:36:31.272680 20045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:36:31.274017 20045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:36:31.275461 20045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:36:31.289532 20050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:31.289621 20053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:31.289587 20051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:36:31.291052 20045 server_base.cc:1034] running on GCE node
I20260926 03:36:31.291623 20045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:36:31.292596 20045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:36:31.293821 20045 hybrid_clock.cc:648] HybridClock initialized: now 1790393791293770 us; error 41 us; skew 500 ppm
I20260926 03:36:31.294138 20045 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:36:31 dist-test-slave-dt7p krb5kdc[19671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:36:31.297431 20045 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:36:31.298887 20045 webserver.cc:460] Webserver started at http://127.0.97.3:43935/ using document root <none> and password file <none>
I20260926 03:36:31.299621 20045 fs_manager.cc:359] Metadata directory not provided
I20260926 03:36:31.299749 20045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:36:31.300081 20045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:36:31.303293 20045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-2/data/instance:
uuid: "af5146c5d70c447f8eaffe394a71e6a7"
format_stamp: "Formatted at 2026-09-26 03:36:31 on dist-test-slave-dt7p"
server_key: "6c4179d004776d05befde7bb04d60415"
server_key_iv: "e2b7132a1d53928f6934ed5652b843c0"
server_key_version: "encryptionkey@0"
I20260926 03:36:31.304157 20045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-2/wal/instance:
uuid: "af5146c5d70c447f8eaffe394a71e6a7"
format_stamp: "Formatted at 2026-09-26 03:36:31 on dist-test-slave-dt7p"
server_key: "6c4179d004776d05befde7bb04d60415"
server_key_iv: "e2b7132a1d53928f6934ed5652b843c0"
server_key_version: "encryptionkey@0"
I20260926 03:36:31.309520 20045 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 03:36:31.313014 20060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:31.314276 20045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:36:31.314445 20045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-2/wal
uuid: "af5146c5d70c447f8eaffe394a71e6a7"
format_stamp: "Formatted at 2026-09-26 03:36:31 on dist-test-slave-dt7p"
server_key: "6c4179d004776d05befde7bb04d60415"
server_key_iv: "e2b7132a1d53928f6934ed5652b843c0"
server_key_version: "encryptionkey@0"
I20260926 03:36:31.314644 20045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:36:31.351065 20045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:36:31.355379 20045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:36:31.355686 20045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:36:31.356820 20045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:36:31.358299 20045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:36:31.358400 20045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:31.358551 20045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:36:31.358623 20045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:31.394945 20045 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34617
I20260926 03:36:31.394989 20172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34617 every 8 connection(s)
I20260926 03:36:31.396458 20045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:36:31.400353 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 20045
I20260926 03:36:31.400596 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:36:31.401135 388 external_mini_cluster.cc:1428] Setting key 466b53fa2e5d472f94d7cd912efc2e3f
Sep 26 03:36:31 dist-test-slave-dt7p krb5kdc[19671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:36:31.420207 20175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:31.399418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50987 (local address 127.0.97.62:38645)
0926 03:36:31.399995 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:31.400002 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:31.401126 (+ 1124us) server_negotiation.cc:407] Connection header received
0926 03:36:31.403819 (+ 2693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:31.403824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:31.403930 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:31.404113 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:31.405972 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:31.406925 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:31.408786 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:31.409272 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:31.409415 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:31.413424 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:31.413430 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:31.413434 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:36:31.413472 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:31.416842 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:31.417590 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:31.417599 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:31.417603 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:31.417745 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:31.418236 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:31.418247 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:31.418249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:31.418557 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:31.418740 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:31.419471 (+ 731us) server_negotiation.cc:299] Negotiation successful
0926 03:36:31.419720 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":259,"threads_started":1}
I20260926 03:36:31.421241 20176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:31.400050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38645 (local address 127.0.97.3:50987)
0926 03:36:31.400934 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:31.400990 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:31.403513 (+ 2523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:31.404395 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:31.404405 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:31.405100 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:36:31.405785 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:31.405806 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:31.407100 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:31.407105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:31.408638 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:31.408653 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:31.409540 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:31.409550 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:31.409850 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:31.410637 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:36:31.410668 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:36:31.413182 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:36:31.416981 (+ 3799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:36:31.416992 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:36:31.417007 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:36:31.417452 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:36:31.417884 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:36:31.417889 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:36:31.417894 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:36:31.418103 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:36:31.418699 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:36:31.418709 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:36:31.419058 (+ 349us) client_negotiation.cc:769] Sending connection context
0926 03:36:31.419344 (+ 286us) client_negotiation.cc:240] Negotiation successful
0926 03:36:31.419898 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":369,"threads_started":1}
I20260926 03:36:31.422644 20173 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38645
I20260926 03:36:31.422996 20173 heartbeater.cc:464] Registering TS with master...
I20260926 03:36:31.423852 20173 heartbeater.cc:511] Master 127.0.97.62:38645 requested a full tablet report, sending...
I20260926 03:36:31.425570 19716 ts_manager.cc:194] Registered new tserver with Master: af5146c5d70c447f8eaffe394a71e6a7 (127.0.97.3:34617)
I20260926 03:36:31.426548 19716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50987
I20260926 03:36:31.428529 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:36:31.445901 20175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:31.433035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59318 (local address 127.0.97.62:38645)
0926 03:36:31.433212 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:31.433221 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:31.433673 (+ 452us) server_negotiation.cc:407] Connection header received
0926 03:36:31.433858 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:31.433864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:31.433937 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:31.434034 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:31.435490 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:31.436212 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:31.437557 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:31.437987 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:31.438110 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:31.440523 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:31.440530 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:31.440534 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:36:31.440618 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:31.443225 (+ 2607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:31.443870 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:31.443877 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:31.443880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:31.443997 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:31.444366 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:31.444371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:31.444375 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:31.444768 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:31.445038 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:31.445345 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 03:36:31.445577 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 03:36:31.449728 19716 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59318:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:36:31.457648 19716 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:36:31.492420 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:31.470667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44007 (local address 127.0.0.1:38092)
0926 03:36:31.480470 (+ 9803us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:31.480537 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:31.480798 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:31.484139 (+ 3341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:31.484147 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:31.484180 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:36:31.484727 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:31.484742 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:31.486735 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:31.486741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:31.489230 (+ 2489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:31.489257 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:31.491277 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:31.491284 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:31.491493 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:31.491551 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 03:36:31.491678 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 03:36:31.491899 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9429,"mutex_wait_us":1,"thread_start_us":14424,"threads_started":1}
I20260926 03:36:31.493150 20191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:31.472798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47234 (local address 127.0.97.1:44139)
0926 03:36:31.485023 (+ 12225us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:31.485031 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:31.485061 (+ 30us) server_negotiation.cc:407] Connection header received
0926 03:36:31.485187 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:31.485194 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:31.485417 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:31.485603 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:36:31.486378 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:31.487908 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:31.490233 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:31.491357 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:31.491480 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:31.491726 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:31.492547 (+ 821us) server_negotiation.cc:299] Negotiation successful
0926 03:36:31.492799 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12031,"thread_start_us":11792,"threads_started":1}
I20260926 03:36:31.493441 20188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:31.470903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38092 (local address 127.0.97.2:44007)
0926 03:36:31.483558 (+ 12655us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:31.483567 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:31.483599 (+ 32us) server_negotiation.cc:407] Connection header received
0926 03:36:31.483706 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:31.483713 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:31.483927 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:31.484139 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:36:31.484949 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:31.486592 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:31.489494 (+ 2902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:31.491044 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:31.491184 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:31.491426 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:31.492893 (+ 1467us) server_negotiation.cc:299] Negotiation successful
0926 03:36:31.493088 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12472,"thread_start_us":12503,"threads_started":1}
I20260926 03:36:31.493585 20187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:31.470787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44139 (local address 127.0.0.1:47234)
0926 03:36:31.483208 (+ 12421us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:31.483227 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:31.483355 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:31.485677 (+ 2322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:31.485681 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:31.485706 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:36:31.486178 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:31.486195 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:31.488078 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:31.488084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:31.490070 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:31.490082 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:31.491878 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:31.491883 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:31.492063 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:31.492096 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 03:36:31.492182 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 03:36:31.492317 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12288,"thread_start_us":12142,"threads_started":1}
I20260926 03:36:31.494894 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:31.471227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34617 (local address 127.0.0.1:49308)
0926 03:36:31.485355 (+ 14128us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:31.485373 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:31.485614 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:31.486456 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:31.486460 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:31.486482 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:36:31.486963 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:31.486979 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:31.488996 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:31.489005 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:31.491726 (+ 2721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:31.491739 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:31.494095 (+ 2356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:31.494104 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:31.494282 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:31.494314 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 03:36:31.494405 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 03:36:31.494531 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13998,"thread_start_us":8681,"threads_started":1}
I20260926 03:36:31.495245 20190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:31.472053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49308 (local address 127.0.97.3:34617)
0926 03:36:31.481682 (+ 9629us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:31.481695 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:31.485473 (+ 3778us) server_negotiation.cc:407] Connection header received
0926 03:36:31.485803 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:31.485812 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:31.486058 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:31.486237 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:36:31.487190 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:31.488855 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:31.491925 (+ 3070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:31.493547 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:31.493659 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:31.493920 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:31.494449 (+ 529us) server_negotiation.cc:299] Negotiation successful
0926 03:36:31.494609 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9378,"thread_start_us":9137,"threads_started":1}
I20260926 03:36:31.498112 19840 tablet_service.cc:1467] Processing CreateTablet for tablet f4f30cfae3b64dddbf71355e4bb5d361 (DEFAULT_TABLE table=test-table [id=e6c051783047448d95bdd41a5b7ab3ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:36:31.498112 19974 tablet_service.cc:1467] Processing CreateTablet for tablet f4f30cfae3b64dddbf71355e4bb5d361 (DEFAULT_TABLE table=test-table [id=e6c051783047448d95bdd41a5b7ab3ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:36:31.498610 20108 tablet_service.cc:1467] Processing CreateTablet for tablet f4f30cfae3b64dddbf71355e4bb5d361 (DEFAULT_TABLE table=test-table [id=e6c051783047448d95bdd41a5b7ab3ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:36:31.499835 19840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4f30cfae3b64dddbf71355e4bb5d361. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:36:31.499835 19974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4f30cfae3b64dddbf71355e4bb5d361. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:36:31.499919 20108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4f30cfae3b64dddbf71355e4bb5d361. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:36:31.512324 20192 tablet_bootstrap.cc:492] T f4f30cfae3b64dddbf71355e4bb5d361 P af5146c5d70c447f8eaffe394a71e6a7: Bootstrap starting.
I20260926 03:36:31.516860 20192 tablet_bootstrap.cc:654] T f4f30cfae3b64dddbf71355e4bb5d361 P af5146c5d70c447f8eaffe394a71e6a7: Neither blocks nor log segments found. Creating new log.
I20260926 03:36:31.517347 20193 tablet_bootstrap.cc:492] T f4f30cfae3b64dddbf71355e4bb5d361 P ed01f9ee4431402ea1aaa61b4e99e54b: Bootstrap starting.
I20260926 03:36:31.518610 20192 log.cc:824] T f4f30cfae3b64dddbf71355e4bb5d361 P af5146c5d70c447f8eaffe394a71e6a7: Log is configured to *not* fsync() on all Append() calls
I20260926 03:36:31.521193 20194 tablet_bootstrap.cc:492] T f4f30cfae3b64dddbf71355e4bb5d361 P 698d3939d3d749b48f049d5ccdb157fe: Bootstrap starting.
I20260926 03:36:31.521739 20193 tablet_bootstrap.cc:654] T f4f30cfae3b64dddbf71355e4bb5d361 P ed01f9ee4431402ea1aaa61b4e99e54b: Neither blocks nor log segments found. Creating new log.
I20260926 03:36:31.523332 20193 log.cc:824] T f4f30cfae3b64dddbf71355e4bb5d361 P ed01f9ee4431402ea1aaa61b4e99e54b: Log is configured to *not* fsync() on all Append() calls
I20260926 03:36:31.524825 20192 tablet_bootstrap.cc:492] T f4f30cfae3b64dddbf71355e4bb5d361 P af5146c5d70c447f8eaffe394a71e6a7: No bootstrap required, opened a new log
I20260926 03:36:31.524999 20194 tablet_bootstrap.cc:654] T f4f30cfae3b64dddbf71355e4bb5d361 P 698d3939d3d749b48f049d5ccdb157fe: Neither blocks nor log segments found. Creating new log.
I20260926 03:36:31.525537 20192 ts_tablet_manager.cc:1397] T f4f30cfae3b64dddbf71355e4bb5d361 P af5146c5d70c447f8eaffe394a71e6a7: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20260926 03:36:31.526079 20194 log.cc:824] T f4f30cfae3b64dddbf71355e4bb5d361 P 698d3939d3d749b48f049d5ccdb157fe: Log is configured to *not* fsync() on all Append() calls
I20260926 03:36:31.527163 20193 tablet_bootstrap.cc:492] T f4f30cfae3b64dddbf71355e4bb5d361 P ed01f9ee4431402ea1aaa61b4e99e54b: No bootstrap required, opened a new log
I20260926 03:36:31.527457 20193 ts_tablet_manager.cc:1397] T f4f30cfae3b64dddbf71355e4bb5d361 P ed01f9ee4431402ea1aaa61b4e99e54b: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20260926 03:36:31.528978 20194 tablet_bootstrap.cc:492] T f4f30cfae3b64dddbf71355e4bb5d361 P 698d3939d3d749b48f049d5ccdb157fe: No bootstrap required, opened a new log
I20260926 03:36:31.529227 20194 ts_tablet_manager.cc:1397] T f4f30cfae3b64dddbf71355e4bb5d361 P 698d3939d3d749b48f049d5ccdb157fe: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260926 03:36:31.537129 20194 raft_consensus.cc:348] T f4f30cfae3b64dddbf71355e4bb5d361 P 698d3939d3d749b48f049d5ccdb157fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698d3939d3d749b48f049d5ccdb157fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44007 } } peers { permanent_uuid: "ed01f9ee4431402ea1aaa61b4e99e54b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44139 } } peers { permanent_uuid: "af5146c5d70c447f8eaffe394a71e6a7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34617 } }
I20260926 03:36:31.537551 20194 raft_consensus.cc:374] T f4f30cfae3b64dddbf71355e4bb5d361 P 698d3939d3d749b48f049d5ccdb157fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:36:31.537333 20192 raft_consensus.cc:348] T f4f30cfae3b64dddbf71355e4bb5d361 P af5146c5d70c447f8eaffe394a71e6a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698d3939d3d749b48f049d5ccdb157fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44007 } } peers { permanent_uuid: "ed01f9ee4431402ea1aaa61b4e99e54b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44139 } } peers { permanent_uuid: "af5146c5d70c447f8eaffe394a71e6a7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34617 } }
I20260926 03:36:31.537658 20194 raft_consensus.cc:729] T f4f30cfae3b64dddbf71355e4bb5d361 P 698d3939d3d749b48f049d5ccdb157fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 698d3939d3d749b48f049d5ccdb157fe, State: Initialized, Role: FOLLOWER
I20260926 03:36:31.537768 20192 raft_consensus.cc:374] T f4f30cfae3b64dddbf71355e4bb5d361 P af5146c5d70c447f8eaffe394a71e6a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:36:31.537863 20192 raft_consensus.cc:729] T f4f30cfae3b64dddbf71355e4bb5d361 P af5146c5d70c447f8eaffe394a71e6a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af5146c5d70c447f8eaffe394a71e6a7, State: Initialized, Role: FOLLOWER
I20260926 03:36:31.538407 20194 consensus_queue.cc:260] T f4f30cfae3b64dddbf71355e4bb5d361 P 698d3939d3d749b48f049d5ccdb157fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698d3939d3d749b48f049d5ccdb157fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44007 } } peers { permanent_uuid: "ed01f9ee4431402ea1aaa61b4e99e54b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44139 } } peers { permanent_uuid: "af5146c5d70c447f8eaffe394a71e6a7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34617 } }
I20260926 03:36:31.538465 20192 consensus_queue.cc:260] T f4f30cfae3b64dddbf71355e4bb5d361 P af5146c5d70c447f8eaffe394a71e6a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698d3939d3d749b48f049d5ccdb157fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44007 } } peers { permanent_uuid: "ed01f9ee4431402ea1aaa61b4e99e54b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44139 } } peers { permanent_uuid: "af5146c5d70c447f8eaffe394a71e6a7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34617 } }
I20260926 03:36:31.538422 20193 raft_consensus.cc:348] T f4f30cfae3b64dddbf71355e4bb5d361 P ed01f9ee4431402ea1aaa61b4e99e54b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698d3939d3d749b48f049d5ccdb157fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44007 } } peers { permanent_uuid: "ed01f9ee4431402ea1aaa61b4e99e54b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44139 } } peers { permanent_uuid: "af5146c5d70c447f8eaffe394a71e6a7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34617 } }
I20260926 03:36:31.538837 20193 raft_consensus.cc:374] T f4f30cfae3b64dddbf71355e4bb5d361 P ed01f9ee4431402ea1aaa61b4e99e54b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:36:31.538941 20193 raft_consensus.cc:729] T f4f30cfae3b64dddbf71355e4bb5d361 P ed01f9ee4431402ea1aaa61b4e99e54b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed01f9ee4431402ea1aaa61b4e99e54b, State: Initialized, Role: FOLLOWER
I20260926 03:36:31.539599 20193 consensus_queue.cc:260] T f4f30cfae3b64dddbf71355e4bb5d361 P ed01f9ee4431402ea1aaa61b4e99e54b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698d3939d3d749b48f049d5ccdb157fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44007 } } peers { permanent_uuid: "ed01f9ee4431402ea1aaa61b4e99e54b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44139 } } peers { permanent_uuid: "af5146c5d70c447f8eaffe394a71e6a7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34617 } }
I20260926 03:36:31.540179 20194 ts_tablet_manager.cc:1428] T f4f30cfae3b64dddbf71355e4bb5d361 P 698d3939d3d749b48f049d5ccdb157fe: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260926 03:36:31.540814 20039 heartbeater.cc:503] Master 127.0.97.62:38645 was elected leader, sending a full tablet report...
I20260926 03:36:31.541327 20193 ts_tablet_manager.cc:1428] T f4f30cfae3b64dddbf71355e4bb5d361 P ed01f9ee4431402ea1aaa61b4e99e54b: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260926 03:36:31.546919 20173 heartbeater.cc:503] Master 127.0.97.62:38645 was elected leader, sending a full tablet report...
I20260926 03:36:31.548708 20192 ts_tablet_manager.cc:1428] T f4f30cfae3b64dddbf71355e4bb5d361 P af5146c5d70c447f8eaffe394a71e6a7: Time spent starting tablet: real 0.023s user 0.018s sys 0.001s
W20260926 03:36:31.559092 20040 tablet.cc:2369] T f4f30cfae3b64dddbf71355e4bb5d361 P 698d3939d3d749b48f049d5ccdb157fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:36:31.638412 20198 raft_consensus.cc:482] T f4f30cfae3b64dddbf71355e4bb5d361 P 698d3939d3d749b48f049d5ccdb157fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:36:31.638809 20198 raft_consensus.cc:504] T f4f30cfae3b64dddbf71355e4bb5d361 P 698d3939d3d749b48f049d5ccdb157fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698d3939d3d749b48f049d5ccdb157fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44007 } } peers { permanent_uuid: "ed01f9ee4431402ea1aaa61b4e99e54b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44139 } } peers { permanent_uuid: "af5146c5d70c447f8eaffe394a71e6a7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34617 } }
I20260926 03:36:31.641075 20198 leader_election.cc:288] T f4f30cfae3b64dddbf71355e4bb5d361 P 698d3939d3d749b48f049d5ccdb157fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed01f9ee4431402ea1aaa61b4e99e54b (127.0.97.1:44139), af5146c5d70c447f8eaffe394a71e6a7 (127.0.97.3:34617)
W20260926 03:36:31.648363 20174 tablet.cc:2369] T f4f30cfae3b64dddbf71355e4bb5d361 P af5146c5d70c447f8eaffe394a71e6a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:36:31.648607 20191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:31.641476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57483 (local address 127.0.97.1:44139)
0926 03:36:31.641747 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:31.641757 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:31.642755 (+ 998us) server_negotiation.cc:407] Connection header received
0926 03:36:31.642949 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:31.642952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:31.643022 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:31.643133 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:36:31.643847 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:31.644626 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:31.646602 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:31.647516 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:31.647622 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:31.647748 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:31.648153 (+ 405us) server_negotiation.cc:299] Negotiation successful
0926 03:36:31.648287 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 03:36:31.648728 20201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:31.641464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44139 (local address 127.0.97.2:57483)
0926 03:36:31.642637 (+ 1173us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:31.642653 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:31.642801 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:31.643168 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:31.643172 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:31.643197 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:36:31.643682 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:31.643695 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:31.644761 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:31.644766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:31.646451 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:31.646462 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:31.647793 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:31.647800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:31.647983 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:31.648018 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 03:36:31.648123 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 03:36:31.648252 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1067,"mutex_wait_us":82,"spinlock_wait_cycles":21248,"thread_start_us":604,"threads_started":1}
I20260926 03:36:31.649142 20190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:31.641781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53165 (local address 127.0.97.3:34617)
0926 03:36:31.642056 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:31.642066 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:31.642589 (+ 523us) server_negotiation.cc:407] Connection header received
0926 03:36:31.642888 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:31.642897 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:31.642990 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:31.643064 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:36:31.643756 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:31.644636 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:31.646666 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:31.647610 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:31.647750 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:31.647875 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:31.648215 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 03:36:31.648375 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 03:36:31.649812 20202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:31.641388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34617 (local address 127.0.97.2:53165)
0926 03:36:31.642447 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:31.642472 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:31.642662 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:31.643106 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:31.643114 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:31.643141 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:36:31.643596 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:31.643605 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:31.644753 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:31.644761 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:31.646487 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:31.646499 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:31.647837 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:31.647841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:31.647993 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:31.648042 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 03:36:31.649263 (+ 1221us) client_negotiation.cc:240] Negotiation successful
0926 03:36:31.649404 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"mutex_wait_us":4,"thread_start_us":933,"threads_started":1}
I20260926 03:36:31.649855 19860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f30cfae3b64dddbf71355e4bb5d361" candidate_uuid: "698d3939d3d749b48f049d5ccdb157fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed01f9ee4431402ea1aaa61b4e99e54b" is_pre_election: true
I20260926 03:36:31.650568 19860 raft_consensus.cc:2454] T f4f30cfae3b64dddbf71355e4bb5d361 P ed01f9ee4431402ea1aaa61b4e99e54b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 698d3939d3d749b48f049d5ccdb157fe in term 0.
I20260926 03:36:31.651136 20128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f30cfae3b64dddbf71355e4bb5d361" candidate_uuid: "698d3939d3d749b48f049d5ccdb157fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af5146c5d70c447f8eaffe394a71e6a7" is_pre_election: true
I20260926 03:36:31.651252 19928 leader_election.cc:302] T f4f30cfae3b64dddbf71355e4bb5d361 P 698d3939d3d749b48f049d5ccdb157fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 698d3939d3d749b48f049d5ccdb157fe, ed01f9ee4431402ea1aaa61b4e99e54b; no voters:
I20260926 03:36:31.651718 20198 raft_consensus.cc:2781] T f4f30cfae3b64dddbf71355e4bb5d361 P 698d3939d3d749b48f049d5ccdb157fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:36:31.651700 20128 raft_consensus.cc:2454] T f4f30cfae3b64dddbf71355e4bb5d361 P af5146c5d70c447f8eaffe394a71e6a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 698d3939d3d749b48f049d5ccdb157fe in term 0.
I20260926 03:36:31.651894 20198 raft_consensus.cc:482] T f4f30cfae3b64dddbf71355e4bb5d361 P 698d3939d3d749b48f049d5ccdb157fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:36:31.652019 20198 raft_consensus.cc:3037] T f4f30cfae3b64dddbf71355e4bb5d361 P 698d3939d3d749b48f049d5ccdb157fe [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:36:31.654781 20198 raft_consensus.cc:504] T f4f30cfae3b64dddbf71355e4bb5d361 P 698d3939d3d749b48f049d5ccdb157fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698d3939d3d749b48f049d5ccdb157fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44007 } } peers { permanent_uuid: "ed01f9ee4431402ea1aaa61b4e99e54b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44139 } } peers { permanent_uuid: "af5146c5d70c447f8eaffe394a71e6a7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34617 } }
I20260926 03:36:31.655500 20198 leader_election.cc:288] T f4f30cfae3b64dddbf71355e4bb5d361 P 698d3939d3d749b48f049d5ccdb157fe [CANDIDATE]: Term 1 election: Requested vote from peers ed01f9ee4431402ea1aaa61b4e99e54b (127.0.97.1:44139), af5146c5d70c447f8eaffe394a71e6a7 (127.0.97.3:34617)
I20260926 03:36:31.655864 19860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f30cfae3b64dddbf71355e4bb5d361" candidate_uuid: "698d3939d3d749b48f049d5ccdb157fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed01f9ee4431402ea1aaa61b4e99e54b"
I20260926 03:36:31.656056 19860 raft_consensus.cc:3037] T f4f30cfae3b64dddbf71355e4bb5d361 P ed01f9ee4431402ea1aaa61b4e99e54b [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:36:31.656037 20128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f30cfae3b64dddbf71355e4bb5d361" candidate_uuid: "698d3939d3d749b48f049d5ccdb157fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af5146c5d70c447f8eaffe394a71e6a7"
I20260926 03:36:31.656288 20128 raft_consensus.cc:3037] T f4f30cfae3b64dddbf71355e4bb5d361 P af5146c5d70c447f8eaffe394a71e6a7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:36:31.658540 19860 raft_consensus.cc:2454] T f4f30cfae3b64dddbf71355e4bb5d361 P ed01f9ee4431402ea1aaa61b4e99e54b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 698d3939d3d749b48f049d5ccdb157fe in term 1.
I20260926 03:36:31.658921 19928 leader_election.cc:302] T f4f30cfae3b64dddbf71355e4bb5d361 P 698d3939d3d749b48f049d5ccdb157fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 698d3939d3d749b48f049d5ccdb157fe, ed01f9ee4431402ea1aaa61b4e99e54b; no voters:
I20260926 03:36:31.659075 20128 raft_consensus.cc:2454] T f4f30cfae3b64dddbf71355e4bb5d361 P af5146c5d70c447f8eaffe394a71e6a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 698d3939d3d749b48f049d5ccdb157fe in term 1.
I20260926 03:36:31.659215 20198 raft_consensus.cc:2781] T f4f30cfae3b64dddbf71355e4bb5d361 P 698d3939d3d749b48f049d5ccdb157fe [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:36:31.659742 20198 raft_consensus.cc:686] T f4f30cfae3b64dddbf71355e4bb5d361 P 698d3939d3d749b48f049d5ccdb157fe [term 1 LEADER]: Becoming Leader. State: Replica: 698d3939d3d749b48f049d5ccdb157fe, State: Running, Role: LEADER
I20260926 03:36:31.660270 20198 consensus_queue.cc:237] T f4f30cfae3b64dddbf71355e4bb5d361 P 698d3939d3d749b48f049d5ccdb157fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698d3939d3d749b48f049d5ccdb157fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44007 } } peers { permanent_uuid: "ed01f9ee4431402ea1aaa61b4e99e54b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44139 } } peers { permanent_uuid: "af5146c5d70c447f8eaffe394a71e6a7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34617 } }
I20260926 03:36:31.665561 19716 catalog_manager.cc:5495] T f4f30cfae3b64dddbf71355e4bb5d361 P 698d3939d3d749b48f049d5ccdb157fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 698d3939d3d749b48f049d5ccdb157fe (127.0.97.2). New cstate: current_term: 1 leader_uuid: "698d3939d3d749b48f049d5ccdb157fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "698d3939d3d749b48f049d5ccdb157fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed01f9ee4431402ea1aaa61b4e99e54b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af5146c5d70c447f8eaffe394a71e6a7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34617 } health_report { overall_health: UNKNOWN } } }
I20260926 03:36:31.685842 20188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:31.679725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38106 (local address 127.0.97.2:44007)
0926 03:36:31.679931 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:31.679936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:31.679971 (+ 35us) server_negotiation.cc:407] Connection header received
0926 03:36:31.680152 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:31.680157 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:31.680281 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:31.680414 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:36:31.680914 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:31.681740 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:31.683263 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:31.683717 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:31.683866 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:31.684415 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:36:31.685149 (+ 734us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:36:31.685322 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:31.685386 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 03:36:31.685488 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:36:31.695761 19860 raft_consensus.cc:1261] T f4f30cfae3b64dddbf71355e4bb5d361 P ed01f9ee4431402ea1aaa61b4e99e54b [term 1 FOLLOWER]: Refusing update from remote peer 698d3939d3d749b48f049d5ccdb157fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:36:31.695765 20128 raft_consensus.cc:1261] T f4f30cfae3b64dddbf71355e4bb5d361 P af5146c5d70c447f8eaffe394a71e6a7 [term 1 FOLLOWER]: Refusing update from remote peer 698d3939d3d749b48f049d5ccdb157fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:36:31.696902 20203 consensus_queue.cc:1035] T f4f30cfae3b64dddbf71355e4bb5d361 P 698d3939d3d749b48f049d5ccdb157fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "af5146c5d70c447f8eaffe394a71e6a7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:36:31.697268 20198 consensus_queue.cc:1035] T f4f30cfae3b64dddbf71355e4bb5d361 P 698d3939d3d749b48f049d5ccdb157fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed01f9ee4431402ea1aaa61b4e99e54b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:36:31.710889 20206 mvcc.cc:205] Tried to move back new op lower bound from 7333452970774401024 to 7333452970645303296. Current Snapshot: MvccSnapshot[applied={T|T < 7333452970774401024}]
I20260926 03:36:31.726588 19715 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:36:31.726927 19715 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:36:31.731017 19715 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b57b5f445be431c894d826860f4c9d7: Sending DeleteTablet for 3 replicas of tablet f4f30cfae3b64dddbf71355e4bb5d361
I20260926 03:36:31.732292 19840 tablet_service.cc:1514] Processing DeleteTablet for tablet f4f30cfae3b64dddbf71355e4bb5d361 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:36:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47234
I20260926 03:36:31.732342 20108 tablet_service.cc:1514] Processing DeleteTablet for tablet f4f30cfae3b64dddbf71355e4bb5d361 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:36:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49308
I20260926 03:36:31.732878 19974 tablet_service.cc:1514] Processing DeleteTablet for tablet f4f30cfae3b64dddbf71355e4bb5d361 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:36:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38092
I20260926 03:36:31.735046 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 19777
I20260926 03:36:31.743472 20218 tablet_replica.cc:330] T f4f30cfae3b64dddbf71355e4bb5d361 P 698d3939d3d749b48f049d5ccdb157fe: stopping tablet replica
I20260926 03:36:31.744737 20218 raft_consensus.cc:2229] T f4f30cfae3b64dddbf71355e4bb5d361 P 698d3939d3d749b48f049d5ccdb157fe [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:36:31.746337 20218 raft_consensus.cc:2258] T f4f30cfae3b64dddbf71355e4bb5d361 P 698d3939d3d749b48f049d5ccdb157fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:36:31.747213 20217 tablet_replica.cc:330] T f4f30cfae3b64dddbf71355e4bb5d361 P af5146c5d70c447f8eaffe394a71e6a7: stopping tablet replica
I20260926 03:36:31.748063 20217 raft_consensus.cc:2229] T f4f30cfae3b64dddbf71355e4bb5d361 P af5146c5d70c447f8eaffe394a71e6a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:36:31.748718 20217 raft_consensus.cc:2258] T f4f30cfae3b64dddbf71355e4bb5d361 P af5146c5d70c447f8eaffe394a71e6a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:36:31.750536 20218 ts_tablet_manager.cc:1905] T f4f30cfae3b64dddbf71355e4bb5d361 P 698d3939d3d749b48f049d5ccdb157fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:36:31.750872 20217 ts_tablet_manager.cc:1905] T f4f30cfae3b64dddbf71355e4bb5d361 P af5146c5d70c447f8eaffe394a71e6a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:36:31.758549 20217 ts_tablet_manager.cc:1918] T f4f30cfae3b64dddbf71355e4bb5d361 P af5146c5d70c447f8eaffe394a71e6a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:36:31.758814 20217 log.cc:1196] T f4f30cfae3b64dddbf71355e4bb5d361 P af5146c5d70c447f8eaffe394a71e6a7: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-2/wal/wals/f4f30cfae3b64dddbf71355e4bb5d361
I20260926 03:36:31.759488 20217 ts_tablet_manager.cc:1939] T f4f30cfae3b64dddbf71355e4bb5d361 P af5146c5d70c447f8eaffe394a71e6a7: Deleting consensus metadata
I20260926 03:36:31.761814 19703 catalog_manager.cc:4841] TS af5146c5d70c447f8eaffe394a71e6a7 (127.0.97.3:34617): tablet f4f30cfae3b64dddbf71355e4bb5d361 (table test-table [id=e6c051783047448d95bdd41a5b7ab3ca]) successfully deleted
I20260926 03:36:31.768342 20216 tablet_replica.cc:330] T f4f30cfae3b64dddbf71355e4bb5d361 P ed01f9ee4431402ea1aaa61b4e99e54b: stopping tablet replica
I20260926 03:36:31.768962 20216 raft_consensus.cc:2229] T f4f30cfae3b64dddbf71355e4bb5d361 P ed01f9ee4431402ea1aaa61b4e99e54b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:36:31.769583 20216 raft_consensus.cc:2258] T f4f30cfae3b64dddbf71355e4bb5d361 P ed01f9ee4431402ea1aaa61b4e99e54b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:36:31.771272 20218 ts_tablet_manager.cc:1918] T f4f30cfae3b64dddbf71355e4bb5d361 P 698d3939d3d749b48f049d5ccdb157fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:36:31.771492 20218 log.cc:1196] T f4f30cfae3b64dddbf71355e4bb5d361 P 698d3939d3d749b48f049d5ccdb157fe: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-1/wal/wals/f4f30cfae3b64dddbf71355e4bb5d361
I20260926 03:36:31.772013 20216 ts_tablet_manager.cc:1905] T f4f30cfae3b64dddbf71355e4bb5d361 P ed01f9ee4431402ea1aaa61b4e99e54b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:36:31.772119 20218 ts_tablet_manager.cc:1939] T f4f30cfae3b64dddbf71355e4bb5d361 P 698d3939d3d749b48f049d5ccdb157fe: Deleting consensus metadata
Sep 26 03:36:31 dist-test-slave-dt7p krb5kdc[19671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:36:31.774209 19700 catalog_manager.cc:4841] TS 698d3939d3d749b48f049d5ccdb157fe (127.0.97.2:44007): tablet f4f30cfae3b64dddbf71355e4bb5d361 (table test-table [id=e6c051783047448d95bdd41a5b7ab3ca]) successfully deleted
I20260926 03:36:31.780006 20216 ts_tablet_manager.cc:1918] T f4f30cfae3b64dddbf71355e4bb5d361 P ed01f9ee4431402ea1aaa61b4e99e54b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:36:31.780201 20216 log.cc:1196] T f4f30cfae3b64dddbf71355e4bb5d361 P ed01f9ee4431402ea1aaa61b4e99e54b: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790393463219689-388-0/minicluster-data/ts-0/wal/wals/f4f30cfae3b64dddbf71355e4bb5d361
I20260926 03:36:31.780654 20191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:31.736815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47246 (local address 127.0.97.1:44139)
0926 03:36:31.737074 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:31.737084 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:31.754542 (+ 17458us) server_negotiation.cc:407] Connection header received
0926 03:36:31.755146 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:31.755156 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:31.755326 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:31.755434 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:31.761414 (+ 5980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:31.763230 (+ 1816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:31.766029 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:31.766797 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:31.766938 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:31.773790 (+ 6852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:31.773811 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:31.773830 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:36:31.773877 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:31.777467 (+ 3590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:31.778153 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:31.778163 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:31.778166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:31.778324 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:31.778709 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:31.778718 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:31.778721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:31.779095 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:31.779445 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:31.779721 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 03:36:31.779983 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 03:36:31.780828 20216 ts_tablet_manager.cc:1939] T f4f30cfae3b64dddbf71355e4bb5d361 P ed01f9ee4431402ea1aaa61b4e99e54b: Deleting consensus metadata
I20260926 03:36:31.781425 19900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:36:31.879226 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 19777
I20260926 03:36:31.901405 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 19911
W20260926 03:36:31.901559 19701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:44139 (error 108)
I20260926 03:36:31.904362 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:31.903712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44139 (local address 127.0.0.1:47252)
0926 03:36:31.904007 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:31.904136 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
W20260926 03:36:31.904624 19701 catalog_manager.cc:4584] TS ed01f9ee4431402ea1aaa61b4e99e54b (127.0.97.1:44139): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f4f30cfae3b64dddbf71355e4bb5d361: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44139: connect: Connection refused (error 111)
Sep 26 03:36:31 dist-test-slave-dt7p krb5kdc[19671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:36:31.922847 20188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:31.903506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38114 (local address 127.0.97.2:44007)
0926 03:36:31.903761 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:31.903770 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:31.905578 (+ 1808us) server_negotiation.cc:407] Connection header received
0926 03:36:31.905848 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:31.905855 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:31.905965 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:31.906085 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:31.908084 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:31.909239 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:31.910616 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:31.911067 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:31.911216 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:31.915975 (+ 4759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:31.915991 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:31.916009 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:36:31.916049 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:31.919919 (+ 3870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:31.920616 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:31.920626 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:31.920630 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:31.920787 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:31.921150 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:31.921155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:31.921159 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:31.921567 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:31.921943 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:31.922232 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 03:36:31.922477 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 03:36:31.923625 20034 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:36:31.958544 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:31.957960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44139 (local address 127.0.0.1:47262)
0926 03:36:31.958224 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:31.958345 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260926 03:36:31.959687 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:31.959298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44139 (local address 127.0.0.1:47270)
0926 03:36:31.959471 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:31.959545 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 03:36:32.020537 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:32.019980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44139 (local address 127.0.0.1:47286)
0926 03:36:32.020226 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:32.020345 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 03:36:32.021723 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:32.021362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44139 (local address 127.0.0.1:47294)
0926 03:36:32.021516 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:32.021598 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 03:36:32.054397 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 19911
I20260926 03:36:32.076519 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 20045
Sep 26 03:36:32 dist-test-slave-dt7p krb5kdc[19671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:36:32.093065 20190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:32.077428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49314 (local address 127.0.97.3:34617)
0926 03:36:32.077634 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:32.077639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:32.077661 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:36:32.077858 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:32.077863 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:32.077942 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:32.078072 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:32.079460 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:32.080302 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:32.081926 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:32.082389 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:32.082492 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:32.087289 (+ 4797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:32.087312 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:32.087333 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:36:32.087387 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:32.090464 (+ 3077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:32.091097 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:32.091105 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:32.091108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:32.091248 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:32.091553 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:32.091556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:32.091558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:32.091944 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:32.092174 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:32.092445 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 03:36:32.092668 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 03:36:32.093786 20168 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:36:32.102593 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:32.102041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44139 (local address 127.0.0.1:47300)
0926 03:36:32.102260 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:32.102370 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 03:36:32.103837 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:32.103361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44139 (local address 127.0.0.1:47304)
0926 03:36:32.103588 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:32.103659 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260926 03:36:32.189507 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 20045
I20260926 03:36:32.208983 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 19684
I20260926 03:36:32.210240 19746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:36:32.337809 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 19684
2026-09-26T03:36:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4704 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T03:36:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:36:32Z Disabled control of system clock
I20260926 03:36:32.426236 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39075
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 03:36:32.804674 20228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:36:32.805037 20228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:36:32.805193 20228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:36:32.812965 20228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:36:32.813086 20228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:36:32.813161 20228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:36:32.813259 20228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:36:32.813333 20228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:36:32.823091 20228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39075
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:36:32.824770 20228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:36:32.826917 20228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:36:32.839907 20233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:32.839989 20234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:32.839956 20236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:36:32.840926 20228 server_base.cc:1034] running on GCE node
I20260926 03:36:32.841715 20228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:36:32.843513 20228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:36:32.844678 20228 hybrid_clock.cc:648] HybridClock initialized: now 1790393792844617 us; error 49 us; skew 500 ppm
I20260926 03:36:32.845026 20228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:36:32.854442 20228 webserver.cc:460] Webserver started at http://127.0.97.62:43093/ using document root <none> and password file <none>
I20260926 03:36:32.855216 20228 fs_manager.cc:359] Metadata directory not provided
I20260926 03:36:32.855306 20228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:36:32.855619 20228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:36:32.858932 20228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "db9af3cc94784ab48436e88f16d12084"
format_stamp: "Formatted at 2026-09-26 03:36:32 on dist-test-slave-dt7p"
server_key: "7c0cb8740b3783183a7bc605cb77e176"
server_key_iv: "7801c1ab31603d847040694c5856e360"
server_key_version: "encryptionkey@0"
I20260926 03:36:32.859767 20228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "db9af3cc94784ab48436e88f16d12084"
format_stamp: "Formatted at 2026-09-26 03:36:32 on dist-test-slave-dt7p"
server_key: "7c0cb8740b3783183a7bc605cb77e176"
server_key_iv: "7801c1ab31603d847040694c5856e360"
server_key_version: "encryptionkey@0"
I20260926 03:36:32.865810 20228 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260926 03:36:32.869670 20242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:32.871179 20228 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:36:32.871336 20228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "db9af3cc94784ab48436e88f16d12084"
format_stamp: "Formatted at 2026-09-26 03:36:32 on dist-test-slave-dt7p"
server_key: "7c0cb8740b3783183a7bc605cb77e176"
server_key_iv: "7801c1ab31603d847040694c5856e360"
server_key_version: "encryptionkey@0"
I20260926 03:36:32.871531 20228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:36:32.885900 20228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:36:32.886814 20228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:36:32.887081 20228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:36:32.907568 20228 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39075
I20260926 03:36:32.907596 20293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39075 every 8 connection(s)
I20260926 03:36:32.909238 20228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:36:32.912135 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 20228
I20260926 03:36:32.912344 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:36:32.913021 388 external_mini_cluster.cc:1428] Setting key 5626925e211da9321051ec2fe15dcb5c
I20260926 03:36:32.914321 20294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:36:32.929741 20297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:32.915167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59766 (local address 127.0.97.62:39075)
0926 03:36:32.916220 (+ 1053us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:32.916232 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:32.916277 (+ 45us) server_negotiation.cc:407] Connection header received
0926 03:36:32.918304 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:32.918335 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:32.919038 (+ 703us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:32.919517 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:32.920135 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:32.922310 (+ 2175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:32.923781 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:32.924880 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:32.925077 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:32.926408 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:32.926420 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:32.926440 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:36:32.927294 (+ 854us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:32.927409 (+ 115us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:36:32.927439 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:32.927561 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:32.927819 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 03:36:32.928235 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":370,"threads_started":1}
I20260926 03:36:32.931085 20294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db9af3cc94784ab48436e88f16d12084: Bootstrap starting.
I20260926 03:36:32.935051 20294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db9af3cc94784ab48436e88f16d12084: Neither blocks nor log segments found. Creating new log.
I20260926 03:36:32.936156 20294 log.cc:824] T 00000000000000000000000000000000 P db9af3cc94784ab48436e88f16d12084: Log is configured to *not* fsync() on all Append() calls
I20260926 03:36:32.939711 20294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db9af3cc94784ab48436e88f16d12084: No bootstrap required, opened a new log
I20260926 03:36:32.948051 20294 raft_consensus.cc:348] T 00000000000000000000000000000000 P db9af3cc94784ab48436e88f16d12084 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9af3cc94784ab48436e88f16d12084" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39075 } }
I20260926 03:36:32.948488 20294 raft_consensus.cc:374] T 00000000000000000000000000000000 P db9af3cc94784ab48436e88f16d12084 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:36:32.948603 20294 raft_consensus.cc:729] T 00000000000000000000000000000000 P db9af3cc94784ab48436e88f16d12084 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db9af3cc94784ab48436e88f16d12084, State: Initialized, Role: FOLLOWER
I20260926 03:36:32.949203 20294 consensus_queue.cc:260] T 00000000000000000000000000000000 P db9af3cc94784ab48436e88f16d12084 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9af3cc94784ab48436e88f16d12084" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39075 } }
I20260926 03:36:32.949470 20294 raft_consensus.cc:388] T 00000000000000000000000000000000 P db9af3cc94784ab48436e88f16d12084 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:36:32.949580 20294 raft_consensus.cc:482] T 00000000000000000000000000000000 P db9af3cc94784ab48436e88f16d12084 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:36:32.949723 20294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db9af3cc94784ab48436e88f16d12084 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:36:32.952528 20294 raft_consensus.cc:504] T 00000000000000000000000000000000 P db9af3cc94784ab48436e88f16d12084 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9af3cc94784ab48436e88f16d12084" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39075 } }
I20260926 03:36:32.953130 20294 leader_election.cc:302] T 00000000000000000000000000000000 P db9af3cc94784ab48436e88f16d12084 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db9af3cc94784ab48436e88f16d12084; no voters:
I20260926 03:36:32.953742 20294 leader_election.cc:288] T 00000000000000000000000000000000 P db9af3cc94784ab48436e88f16d12084 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:36:32.953912 20299 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db9af3cc94784ab48436e88f16d12084 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:36:32.954746 20299 raft_consensus.cc:686] T 00000000000000000000000000000000 P db9af3cc94784ab48436e88f16d12084 [term 1 LEADER]: Becoming Leader. State: Replica: db9af3cc94784ab48436e88f16d12084, State: Running, Role: LEADER
I20260926 03:36:32.955336 20299 consensus_queue.cc:237] T 00000000000000000000000000000000 P db9af3cc94784ab48436e88f16d12084 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9af3cc94784ab48436e88f16d12084" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39075 } }
I20260926 03:36:32.956144 20294 sys_catalog.cc:564] T 00000000000000000000000000000000 P db9af3cc94784ab48436e88f16d12084 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:36:32.958271 20301 sys_catalog.cc:455] T 00000000000000000000000000000000 P db9af3cc94784ab48436e88f16d12084 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db9af3cc94784ab48436e88f16d12084. Latest consensus state: current_term: 1 leader_uuid: "db9af3cc94784ab48436e88f16d12084" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9af3cc94784ab48436e88f16d12084" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39075 } } }
I20260926 03:36:32.958782 20301 sys_catalog.cc:458] T 00000000000000000000000000000000 P db9af3cc94784ab48436e88f16d12084 [sys.catalog]: This master's current role is: LEADER
I20260926 03:36:32.958781 20300 sys_catalog.cc:455] T 00000000000000000000000000000000 P db9af3cc94784ab48436e88f16d12084 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db9af3cc94784ab48436e88f16d12084" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db9af3cc94784ab48436e88f16d12084" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39075 } } }
I20260926 03:36:32.959182 20300 sys_catalog.cc:458] T 00000000000000000000000000000000 P db9af3cc94784ab48436e88f16d12084 [sys.catalog]: This master's current role is: LEADER
I20260926 03:36:32.962900 20305 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:36:32.970336 20305 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:36:32.980934 20305 catalog_manager.cc:1347] Generated new cluster ID: cd7a417c50e4449b86734fede83870ed
I20260926 03:36:32.981057 20305 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:36:33.012045 20305 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:36:33.013837 20305 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:36:33.025008 20305 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db9af3cc94784ab48436e88f16d12084: Generated new TSK 0
I20260926 03:36:33.025996 20305 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:36:33.032148 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39075
--builtin_ntp_servers=127.0.97.20:45421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 03:36:33.422468 20319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:36:33.422798 20319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:36:33.422894 20319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:36:33.431552 20319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:36:33.431670 20319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:36:33.431792 20319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:36:33.442525 20319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:36:33.444110 20319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:36:33.446447 20319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:36:33.461542 20324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:33.461500 20327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:33.461607 20325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:36:33.466228 20319 server_base.cc:1034] running on GCE node
I20260926 03:36:33.467020 20319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:36:33.468078 20319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:36:33.469286 20319 hybrid_clock.cc:648] HybridClock initialized: now 1790393793469233 us; error 47 us; skew 500 ppm
I20260926 03:36:33.469624 20319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:36:33.472210 20319 webserver.cc:460] Webserver started at http://127.0.97.1:44479/ using document root <none> and password file <none>
I20260926 03:36:33.473042 20319 fs_manager.cc:359] Metadata directory not provided
I20260926 03:36:33.473165 20319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:36:33.473519 20319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:36:33.476945 20319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-0/data/instance:
uuid: "a4f19bab68c146fd8fb9f4ff5a086e59"
format_stamp: "Formatted at 2026-09-26 03:36:33 on dist-test-slave-dt7p"
server_key: "2deba86f80cad8689fc49f796595ab5d"
server_key_iv: "bbe1b34d7c7ea79ca358ad79dcf2f856"
server_key_version: "encryptionkey@0"
I20260926 03:36:33.477764 20319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a4f19bab68c146fd8fb9f4ff5a086e59"
format_stamp: "Formatted at 2026-09-26 03:36:33 on dist-test-slave-dt7p"
server_key: "2deba86f80cad8689fc49f796595ab5d"
server_key_iv: "bbe1b34d7c7ea79ca358ad79dcf2f856"
server_key_version: "encryptionkey@0"
I20260926 03:36:33.483166 20319 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:36:33.486975 20333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:33.488382 20319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:36:33.488590 20319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-0/wal
uuid: "a4f19bab68c146fd8fb9f4ff5a086e59"
format_stamp: "Formatted at 2026-09-26 03:36:33 on dist-test-slave-dt7p"
server_key: "2deba86f80cad8689fc49f796595ab5d"
server_key_iv: "bbe1b34d7c7ea79ca358ad79dcf2f856"
server_key_version: "encryptionkey@0"
I20260926 03:36:33.488827 20319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:36:33.512810 20319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:36:33.513775 20319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:36:33.514081 20319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:36:33.515158 20319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:36:33.516934 20319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:36:33.517066 20319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:33.517194 20319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:36:33.517285 20319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:33.553260 20319 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41317
I20260926 03:36:33.553323 20445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41317 every 8 connection(s)
I20260926 03:36:33.554797 20319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:36:33.558020 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 20319
I20260926 03:36:33.558207 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:36:33.558823 388 external_mini_cluster.cc:1428] Setting key 07c18245aae0f242b5eeb5534fbf8177
I20260926 03:36:33.562013 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39075
--builtin_ntp_servers=127.0.97.20:45421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 03:36:33.569201 20448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:33.557834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57067 (local address 127.0.97.62:39075)
0926 03:36:33.558406 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:33.558411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:33.559868 (+ 1457us) server_negotiation.cc:407] Connection header received
0926 03:36:33.562058 (+ 2190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:33.562064 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:33.562165 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:33.562284 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:33.563471 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:33.564270 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:33.566055 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:33.566526 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:33.566664 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:33.568214 (+ 1550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:33.568226 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:33.568229 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:36:33.568250 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:33.568297 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:36:33.568312 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:33.568402 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:33.568680 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 03:36:33.568805 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":286,"threads_started":1}
I20260926 03:36:33.570310 20449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:33.558550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39075 (local address 127.0.97.1:57067)
0926 03:36:33.559684 (+ 1134us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:33.559744 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:33.561779 (+ 2035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:33.562549 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:33.562558 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:33.562657 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:36:33.563292 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:33.563319 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:33.564426 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:33.564432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:33.565923 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:33.565936 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:33.566768 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:33.566777 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:33.567089 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:33.567883 (+ 794us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:36:33.567886 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:36:33.567931 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:36:33.568018 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:36:33.568413 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:36:33.568427 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:36:33.568436 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 03:36:33.568664 (+ 228us) client_negotiation.cc:240] Negotiation successful
0926 03:36:33.569094 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":355,"threads_started":1}
I20260926 03:36:33.572109 20446 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39075
I20260926 03:36:33.572443 20446 heartbeater.cc:464] Registering TS with master...
I20260926 03:36:33.573331 20446 heartbeater.cc:511] Master 127.0.97.62:39075 requested a full tablet report, sending...
I20260926 03:36:33.575876 20259 ts_manager.cc:194] Registered new tserver with Master: a4f19bab68c146fd8fb9f4ff5a086e59 (127.0.97.1:41317)
I20260926 03:36:33.578235 20259 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:57067
W20260926 03:36:33.953716 20450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:36:33.954054 20450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:36:33.954263 20450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:36:33.962448 20450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:36:33.962584 20450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:36:33.962772 20450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:36:33.972697 20450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:36:33.974455 20450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:36:33.976763 20450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:36:33.991158 20455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:33.991112 20458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:33.991190 20456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:36:33.991882 20450 server_base.cc:1034] running on GCE node
I20260926 03:36:33.992655 20450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:36:33.993713 20450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:36:33.994944 20450 hybrid_clock.cc:648] HybridClock initialized: now 1790393793994907 us; error 49 us; skew 500 ppm
I20260926 03:36:33.995275 20450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:36:33.997750 20450 webserver.cc:460] Webserver started at http://127.0.97.2:38779/ using document root <none> and password file <none>
I20260926 03:36:33.998518 20450 fs_manager.cc:359] Metadata directory not provided
I20260926 03:36:33.998651 20450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:36:33.999010 20450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:36:34.002177 20450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-1/data/instance:
uuid: "e2c16ecf9d7b4ee4b23b2a6b1258e75a"
format_stamp: "Formatted at 2026-09-26 03:36:33 on dist-test-slave-dt7p"
server_key: "b2696e9dc41e5e3bdaeb6c77a44db1ba"
server_key_iv: "3c530002084e196716f124fb0e0239d5"
server_key_version: "encryptionkey@0"
I20260926 03:36:34.003028 20450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e2c16ecf9d7b4ee4b23b2a6b1258e75a"
format_stamp: "Formatted at 2026-09-26 03:36:33 on dist-test-slave-dt7p"
server_key: "b2696e9dc41e5e3bdaeb6c77a44db1ba"
server_key_iv: "3c530002084e196716f124fb0e0239d5"
server_key_version: "encryptionkey@0"
I20260926 03:36:34.008289 20450 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260926 03:36:34.011857 20464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:34.013326 20450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:36:34.013504 20450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-1/wal
uuid: "e2c16ecf9d7b4ee4b23b2a6b1258e75a"
format_stamp: "Formatted at 2026-09-26 03:36:33 on dist-test-slave-dt7p"
server_key: "b2696e9dc41e5e3bdaeb6c77a44db1ba"
server_key_iv: "3c530002084e196716f124fb0e0239d5"
server_key_version: "encryptionkey@0"
I20260926 03:36:34.013729 20450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:36:34.041714 20450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:36:34.042765 20450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:36:34.043081 20450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:36:34.044215 20450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:36:34.046000 20450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:36:34.046134 20450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:34.046267 20450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:36:34.046351 20450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:34.082324 20450 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44391
I20260926 03:36:34.082415 20576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44391 every 8 connection(s)
I20260926 03:36:34.083787 20450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:36:34.084302 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 20450
I20260926 03:36:34.084501 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:36:34.085260 388 external_mini_cluster.cc:1428] Setting key 984344b7ee347411f0c1465d8e679b90
I20260926 03:36:34.089084 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39075
--builtin_ntp_servers=127.0.97.20:45421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 03:36:34.100384 20579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:34.086883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49863 (local address 127.0.97.62:39075)
0926 03:36:34.087544 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:34.087553 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:34.089089 (+ 1536us) server_negotiation.cc:407] Connection header received
0926 03:36:34.092253 (+ 3164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:34.092257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:34.092336 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:34.092432 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:34.093950 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:34.094761 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:34.097330 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:34.097765 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:34.097876 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:34.099572 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:34.099578 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:34.099581 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:36:34.099603 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:34.099653 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:36:34.099666 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:34.099733 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:34.099996 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 03:36:34.100088 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":351,"threads_started":1}
I20260926 03:36:34.101826 20580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:34.087638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39075 (local address 127.0.97.2:49863)
0926 03:36:34.088906 (+ 1268us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:34.088963 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:34.091919 (+ 2956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:34.092763 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:34.092777 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:34.092903 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:36:34.093767 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:34.093799 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:34.094925 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:34.094935 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:34.097181 (+ 2246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:34.097199 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:34.098003 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:34.098012 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:34.098338 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:34.099187 (+ 849us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:36:34.099193 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:36:34.099257 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:36:34.099365 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:36:34.099754 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:36:34.099767 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:36:34.099778 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 03:36:34.099968 (+ 190us) client_negotiation.cc:240] Negotiation successful
0926 03:36:34.100415 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":375,"threads_started":1}
I20260926 03:36:34.103422 20577 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39075
I20260926 03:36:34.103756 20577 heartbeater.cc:464] Registering TS with master...
I20260926 03:36:34.104601 20577 heartbeater.cc:511] Master 127.0.97.62:39075 requested a full tablet report, sending...
I20260926 03:36:34.106257 20259 ts_manager.cc:194] Registered new tserver with Master: e2c16ecf9d7b4ee4b23b2a6b1258e75a (127.0.97.2:44391)
I20260926 03:36:34.107170 20259 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:49863
W20260926 03:36:34.469357 20581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:36:34.469699 20581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:36:34.469858 20581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:36:34.478055 20581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:36:34.478191 20581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:36:34.478375 20581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:36:34.487849 20581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:36:34.489478 20581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:36:34.491607 20581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:36:34.506510 20586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:34.506623 20589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:34.506656 20587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:36:34.507645 20581 server_base.cc:1034] running on GCE node
I20260926 03:36:34.508345 20581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:36:34.509409 20581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:36:34.510645 20581 hybrid_clock.cc:648] HybridClock initialized: now 1790393794510580 us; error 56 us; skew 500 ppm
I20260926 03:36:34.510991 20581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:36:34.513574 20581 webserver.cc:460] Webserver started at http://127.0.97.3:45801/ using document root <none> and password file <none>
I20260926 03:36:34.514398 20581 fs_manager.cc:359] Metadata directory not provided
I20260926 03:36:34.514530 20581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:36:34.514889 20581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:36:34.518250 20581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-2/data/instance:
uuid: "f10b3a87de534c3da57c237357160083"
format_stamp: "Formatted at 2026-09-26 03:36:34 on dist-test-slave-dt7p"
server_key: "fec588e13000c6027814d69632e167c8"
server_key_iv: "e3d70d94e228de4ec366f38349afb9c2"
server_key_version: "encryptionkey@0"
I20260926 03:36:34.519186 20581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f10b3a87de534c3da57c237357160083"
format_stamp: "Formatted at 2026-09-26 03:36:34 on dist-test-slave-dt7p"
server_key: "fec588e13000c6027814d69632e167c8"
server_key_iv: "e3d70d94e228de4ec366f38349afb9c2"
server_key_version: "encryptionkey@0"
I20260926 03:36:34.525292 20581 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260926 03:36:34.529426 20595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:34.531073 20581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20260926 03:36:34.531302 20581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-2/wal
uuid: "f10b3a87de534c3da57c237357160083"
format_stamp: "Formatted at 2026-09-26 03:36:34 on dist-test-slave-dt7p"
server_key: "fec588e13000c6027814d69632e167c8"
server_key_iv: "e3d70d94e228de4ec366f38349afb9c2"
server_key_version: "encryptionkey@0"
I20260926 03:36:34.531597 20581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:36:34.559873 20581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:36:34.560842 20581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:36:34.561089 20581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:36:34.562071 20581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:36:34.563659 20581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:36:34.563748 20581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:34.563836 20581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:36:34.563894 20581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:34.581727 20446 heartbeater.cc:503] Master 127.0.97.62:39075 was elected leader, sending a full tablet report...
I20260926 03:36:34.600378 20581 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:32893
I20260926 03:36:34.600431 20707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:32893 every 8 connection(s)
I20260926 03:36:34.601963 20581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:36:34.602758 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 20581
I20260926 03:36:34.602979 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:36:34.603682 388 external_mini_cluster.cc:1428] Setting key d4efa2cb1a2aec28523efcbc18cb4de2
I20260926 03:36:34.616253 20710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:34.605124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36735 (local address 127.0.97.62:39075)
0926 03:36:34.605622 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:34.605629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:34.607321 (+ 1692us) server_negotiation.cc:407] Connection header received
0926 03:36:34.609346 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:34.609351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:34.609448 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:34.609574 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:34.610746 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:34.611645 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:34.613387 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:34.613815 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:34.613942 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:34.615320 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:34.615328 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:34.615333 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:36:34.615358 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:34.615409 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:36:34.615422 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:34.615512 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:34.615730 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 03:36:34.615837 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":218,"threads_started":1}
I20260926 03:36:34.617336 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:34.605953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39075 (local address 127.0.97.3:36735)
0926 03:36:34.607142 (+ 1189us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:34.607199 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:34.609063 (+ 1864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:34.609822 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:34.609831 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:34.609922 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:36:34.610571 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:34.610594 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:34.611807 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:34.611813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:34.613248 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:34.613261 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:34.614044 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:34.614051 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:34.614288 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:34.614976 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:36:34.614979 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:36:34.615026 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:36:34.615101 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:36:34.615510 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:36:34.615522 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:36:34.615532 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 03:36:34.615717 (+ 185us) client_negotiation.cc:240] Negotiation successful
0926 03:36:34.616104 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":371,"threads_started":1}
I20260926 03:36:34.618884 20708 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39075
I20260926 03:36:34.619222 20708 heartbeater.cc:464] Registering TS with master...
I20260926 03:36:34.620092 20708 heartbeater.cc:511] Master 127.0.97.62:39075 requested a full tablet report, sending...
I20260926 03:36:34.621645 20258 ts_manager.cc:194] Registered new tserver with Master: f10b3a87de534c3da57c237357160083 (127.0.97.3:32893)
I20260926 03:36:34.622658 20258 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:36735
I20260926 03:36:34.632081 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:36:34.643013 20710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:34.636805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59770 (local address 127.0.97.62:39075)
0926 03:36:34.637052 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:34.637062 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:34.637446 (+ 384us) server_negotiation.cc:407] Connection header received
0926 03:36:34.637606 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:34.637611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:34.637688 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:34.637771 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:34.638359 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:34.639118 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:34.640652 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:34.641068 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:34.641177 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:34.642255 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:34.642260 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:34.642263 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:36:34.642281 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:34.642316 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:36:34.642328 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:34.642397 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:34.642553 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 03:36:34.642634 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:36:34.647174 20258 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59770:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:36:34.654825 20258 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:36:34.682215 20721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:34.668267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44391 (local address 127.0.0.1:47502)
0926 03:36:34.669492 (+ 1225us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:34.669542 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:34.669732 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:34.671127 (+ 1395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:34.671139 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:34.671195 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:36:34.671760 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:34.671777 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:34.674602 (+ 2825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:34.674611 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:34.677973 (+ 3362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:34.677995 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:34.681112 (+ 3117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:34.681123 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:34.681340 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:34.681401 (+ 61us) client_negotiation.cc:769] Sending connection context
0926 03:36:34.681561 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 03:36:34.681766 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":917,"thread_start_us":1125,"threads_started":1}
I20260926 03:36:34.683418 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:34.668988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47502 (local address 127.0.97.2:44391)
0926 03:36:34.669678 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:34.669692 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:34.669741 (+ 49us) server_negotiation.cc:407] Connection header received
0926 03:36:34.670001 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:34.670014 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:34.670375 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:34.670683 (+ 308us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:36:34.672044 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:34.674378 (+ 2334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:34.678289 (+ 3911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:34.680442 (+ 2153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:34.680657 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:34.680986 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:34.682710 (+ 1724us) server_negotiation.cc:299] Negotiation successful
0926 03:36:34.682981 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":323,"threads_started":1}
I20260926 03:36:34.689229 20512 tablet_service.cc:1467] Processing CreateTablet for tablet 4324adc40dcc42fd8ca8db2c7a5d2ec3 (DEFAULT_TABLE table=test-table [id=3a21d31a0ed4421a948b5a60146c4da7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:36:34.690900 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:34.668541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:32893 (local address 127.0.0.1:43836)
0926 03:36:34.681154 (+ 12613us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:34.681179 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:34.681366 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:34.682129 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:34.682137 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:34.682178 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:36:34.682611 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:34.682622 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:34.685124 (+ 2502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:34.685129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:34.687510 (+ 2381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:34.687526 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:34.689489 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:34.689495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:34.689688 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:34.689722 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 03:36:34.689823 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 03:36:34.689977 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12428,"mutex_wait_us":2,"thread_start_us":12227,"threads_started":1}
I20260926 03:36:34.691172 20512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4324adc40dcc42fd8ca8db2c7a5d2ec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:36:34.691386 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:34.671161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43316 (local address 127.0.97.1:41317)
0926 03:36:34.682029 (+ 10868us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:34.682041 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:34.682882 (+ 841us) server_negotiation.cc:407] Connection header received
0926 03:36:34.683016 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:34.683024 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:34.683341 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:34.683574 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:36:34.684312 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:34.685875 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:34.688130 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:34.689670 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:34.689809 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:34.690672 (+ 863us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:34.690757 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 03:36:34.690937 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10647,"thread_start_us":10468,"threads_started":1}
I20260926 03:36:34.692018 20724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:34.669035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43836 (local address 127.0.97.3:32893)
0926 03:36:34.680351 (+ 11316us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:34.680364 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:34.681324 (+ 960us) server_negotiation.cc:407] Connection header received
0926 03:36:34.681577 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:34.681586 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:34.681899 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:34.682144 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:36:34.683275 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:34.684965 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:34.687759 (+ 2794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:34.689258 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:34.689412 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:34.689728 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:34.691398 (+ 1670us) server_negotiation.cc:299] Negotiation successful
0926 03:36:34.691711 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11043,"thread_start_us":11314,"threads_started":1}
I20260926 03:36:34.692329 20721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:34.670323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41317 (local address 127.0.0.1:43316)
0926 03:36:34.682540 (+ 12217us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:34.682563 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:34.682671 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:34.683849 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:34.683853 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:34.683879 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:36:34.684104 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:34.684114 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:34.686037 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:34.686042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:34.687964 (+ 1922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:34.687979 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:34.689851 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:34.689856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:34.690070 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:34.690106 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 03:36:34.690187 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 03:36:34.690339 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12107,"mutex_wait_us":3,"thread_start_us":12617,"threads_started":1}
I20260926 03:36:34.696305 20381 tablet_service.cc:1467] Processing CreateTablet for tablet 4324adc40dcc42fd8ca8db2c7a5d2ec3 (DEFAULT_TABLE table=test-table [id=3a21d31a0ed4421a948b5a60146c4da7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:36:34.698122 20381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4324adc40dcc42fd8ca8db2c7a5d2ec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:36:34.699458 20643 tablet_service.cc:1467] Processing CreateTablet for tablet 4324adc40dcc42fd8ca8db2c7a5d2ec3 (DEFAULT_TABLE table=test-table [id=3a21d31a0ed4421a948b5a60146c4da7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:36:34.701475 20643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4324adc40dcc42fd8ca8db2c7a5d2ec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:36:34.704667 20727 tablet_bootstrap.cc:492] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P e2c16ecf9d7b4ee4b23b2a6b1258e75a: Bootstrap starting.
I20260926 03:36:34.710222 20727 tablet_bootstrap.cc:654] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P e2c16ecf9d7b4ee4b23b2a6b1258e75a: Neither blocks nor log segments found. Creating new log.
I20260926 03:36:34.711354 20727 log.cc:824] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P e2c16ecf9d7b4ee4b23b2a6b1258e75a: Log is configured to *not* fsync() on all Append() calls
I20260926 03:36:34.715209 20727 tablet_bootstrap.cc:492] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P e2c16ecf9d7b4ee4b23b2a6b1258e75a: No bootstrap required, opened a new log
I20260926 03:36:34.715224 20729 tablet_bootstrap.cc:492] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P a4f19bab68c146fd8fb9f4ff5a086e59: Bootstrap starting.
I20260926 03:36:34.715528 20727 ts_tablet_manager.cc:1397] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P e2c16ecf9d7b4ee4b23b2a6b1258e75a: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260926 03:36:34.716342 20730 tablet_bootstrap.cc:492] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P f10b3a87de534c3da57c237357160083: Bootstrap starting.
I20260926 03:36:34.719239 20729 tablet_bootstrap.cc:654] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P a4f19bab68c146fd8fb9f4ff5a086e59: Neither blocks nor log segments found. Creating new log.
I20260926 03:36:34.719770 20730 tablet_bootstrap.cc:654] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P f10b3a87de534c3da57c237357160083: Neither blocks nor log segments found. Creating new log.
I20260926 03:36:34.720667 20729 log.cc:824] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P a4f19bab68c146fd8fb9f4ff5a086e59: Log is configured to *not* fsync() on all Append() calls
I20260926 03:36:34.720824 20730 log.cc:824] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P f10b3a87de534c3da57c237357160083: Log is configured to *not* fsync() on all Append() calls
I20260926 03:36:34.723908 20730 tablet_bootstrap.cc:492] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P f10b3a87de534c3da57c237357160083: No bootstrap required, opened a new log
I20260926 03:36:34.723908 20729 tablet_bootstrap.cc:492] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P a4f19bab68c146fd8fb9f4ff5a086e59: No bootstrap required, opened a new log
I20260926 03:36:34.724197 20729 ts_tablet_manager.cc:1397] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P a4f19bab68c146fd8fb9f4ff5a086e59: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260926 03:36:34.724185 20730 ts_tablet_manager.cc:1397] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P f10b3a87de534c3da57c237357160083: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260926 03:36:34.726239 20727 raft_consensus.cc:348] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P e2c16ecf9d7b4ee4b23b2a6b1258e75a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c16ecf9d7b4ee4b23b2a6b1258e75a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44391 } } peers { permanent_uuid: "f10b3a87de534c3da57c237357160083" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32893 } } peers { permanent_uuid: "a4f19bab68c146fd8fb9f4ff5a086e59" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41317 } }
I20260926 03:36:34.726643 20727 raft_consensus.cc:374] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P e2c16ecf9d7b4ee4b23b2a6b1258e75a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:36:34.726745 20727 raft_consensus.cc:729] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P e2c16ecf9d7b4ee4b23b2a6b1258e75a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c16ecf9d7b4ee4b23b2a6b1258e75a, State: Initialized, Role: FOLLOWER
I20260926 03:36:34.727389 20727 consensus_queue.cc:260] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P e2c16ecf9d7b4ee4b23b2a6b1258e75a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c16ecf9d7b4ee4b23b2a6b1258e75a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44391 } } peers { permanent_uuid: "f10b3a87de534c3da57c237357160083" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32893 } } peers { permanent_uuid: "a4f19bab68c146fd8fb9f4ff5a086e59" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41317 } }
I20260926 03:36:34.728554 20577 heartbeater.cc:503] Master 127.0.97.62:39075 was elected leader, sending a full tablet report...
I20260926 03:36:34.730701 20727 ts_tablet_manager.cc:1428] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P e2c16ecf9d7b4ee4b23b2a6b1258e75a: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260926 03:36:34.735450 20730 raft_consensus.cc:348] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P f10b3a87de534c3da57c237357160083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c16ecf9d7b4ee4b23b2a6b1258e75a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44391 } } peers { permanent_uuid: "f10b3a87de534c3da57c237357160083" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32893 } } peers { permanent_uuid: "a4f19bab68c146fd8fb9f4ff5a086e59" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41317 } }
I20260926 03:36:34.735468 20729 raft_consensus.cc:348] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P a4f19bab68c146fd8fb9f4ff5a086e59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c16ecf9d7b4ee4b23b2a6b1258e75a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44391 } } peers { permanent_uuid: "f10b3a87de534c3da57c237357160083" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32893 } } peers { permanent_uuid: "a4f19bab68c146fd8fb9f4ff5a086e59" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41317 } }
I20260926 03:36:34.735925 20730 raft_consensus.cc:374] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P f10b3a87de534c3da57c237357160083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:36:34.735973 20729 raft_consensus.cc:374] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P a4f19bab68c146fd8fb9f4ff5a086e59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:36:34.736032 20730 raft_consensus.cc:729] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P f10b3a87de534c3da57c237357160083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f10b3a87de534c3da57c237357160083, State: Initialized, Role: FOLLOWER
I20260926 03:36:34.736068 20729 raft_consensus.cc:729] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P a4f19bab68c146fd8fb9f4ff5a086e59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4f19bab68c146fd8fb9f4ff5a086e59, State: Initialized, Role: FOLLOWER
I20260926 03:36:34.736724 20730 consensus_queue.cc:260] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P f10b3a87de534c3da57c237357160083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c16ecf9d7b4ee4b23b2a6b1258e75a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44391 } } peers { permanent_uuid: "f10b3a87de534c3da57c237357160083" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32893 } } peers { permanent_uuid: "a4f19bab68c146fd8fb9f4ff5a086e59" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41317 } }
I20260926 03:36:34.736718 20729 consensus_queue.cc:260] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P a4f19bab68c146fd8fb9f4ff5a086e59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c16ecf9d7b4ee4b23b2a6b1258e75a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44391 } } peers { permanent_uuid: "f10b3a87de534c3da57c237357160083" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32893 } } peers { permanent_uuid: "a4f19bab68c146fd8fb9f4ff5a086e59" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41317 } }
I20260926 03:36:34.738034 20708 heartbeater.cc:503] Master 127.0.97.62:39075 was elected leader, sending a full tablet report...
I20260926 03:36:34.738617 20730 ts_tablet_manager.cc:1428] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P f10b3a87de534c3da57c237357160083: Time spent starting tablet: real 0.014s user 0.015s sys 0.001s
I20260926 03:36:34.738641 20729 ts_tablet_manager.cc:1428] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P a4f19bab68c146fd8fb9f4ff5a086e59: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
W20260926 03:36:34.807797 20447 tablet.cc:2369] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P a4f19bab68c146fd8fb9f4ff5a086e59: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:36:34.836292 20578 tablet.cc:2369] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P e2c16ecf9d7b4ee4b23b2a6b1258e75a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:36:34.854027 20709 tablet.cc:2369] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P f10b3a87de534c3da57c237357160083: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:36:35.054481 20733 raft_consensus.cc:482] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P e2c16ecf9d7b4ee4b23b2a6b1258e75a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:36:35.054885 20733 raft_consensus.cc:504] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P e2c16ecf9d7b4ee4b23b2a6b1258e75a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c16ecf9d7b4ee4b23b2a6b1258e75a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44391 } } peers { permanent_uuid: "f10b3a87de534c3da57c237357160083" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32893 } } peers { permanent_uuid: "a4f19bab68c146fd8fb9f4ff5a086e59" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41317 } }
I20260926 03:36:35.057134 20733 leader_election.cc:288] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P e2c16ecf9d7b4ee4b23b2a6b1258e75a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f10b3a87de534c3da57c237357160083 (127.0.97.3:32893), a4f19bab68c146fd8fb9f4ff5a086e59 (127.0.97.1:41317)
I20260926 03:36:35.064605 20724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:35.057676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33541 (local address 127.0.97.3:32893)
0926 03:36:35.057953 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:35.057962 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:35.058567 (+ 605us) server_negotiation.cc:407] Connection header received
0926 03:36:35.058898 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:35.058907 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:35.059017 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:35.059123 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:36:35.059854 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:35.060920 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:35.062822 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:35.063621 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:35.063781 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:35.063903 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:35.064167 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 03:36:35.064334 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 03:36:35.064642 20736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:35.057573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:32893 (local address 127.0.97.2:33541)
0926 03:36:35.058449 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:35.058473 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:35.058713 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:35.059166 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:35.059177 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:35.059215 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:36:35.059702 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:35.059715 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:35.061048 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:35.061052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:35.062615 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:35.062627 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:35.063828 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:35.063834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:35.063996 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:35.064021 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 03:36:35.064130 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 03:36:35.064258 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":453,"threads_started":1}
I20260926 03:36:35.064889 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:35.057671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40985 (local address 127.0.97.1:41317)
0926 03:36:35.058156 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:35.058164 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:35.059512 (+ 1348us) server_negotiation.cc:407] Connection header received
0926 03:36:35.059726 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:35.059730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:35.059838 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:35.059931 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:36:35.060457 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:35.061395 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:35.063099 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:35.063901 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:35.064050 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:35.064167 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:35.064410 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 03:36:35.064552 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316}
I20260926 03:36:35.065331 20737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:35.058528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41317 (local address 127.0.97.2:40985)
0926 03:36:35.059404 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0926 03:36:35.059419 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:36:35.059562 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:36:35.059954 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:36:35.059957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:36:35.059976 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:36:35.060297 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:35.060304 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:35.061529 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:35.061535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:35.062927 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:36:35.062939 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:36:35.064083 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:36:35.064089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:36:35.064262 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:35.064285 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 03:36:35.064380 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 03:36:35.064496 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"thread_start_us":440,"threads_started":1}
I20260926 03:36:35.066226 20663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4324adc40dcc42fd8ca8db2c7a5d2ec3" candidate_uuid: "e2c16ecf9d7b4ee4b23b2a6b1258e75a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f10b3a87de534c3da57c237357160083" is_pre_election: true
I20260926 03:36:35.066498 20401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4324adc40dcc42fd8ca8db2c7a5d2ec3" candidate_uuid: "e2c16ecf9d7b4ee4b23b2a6b1258e75a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4f19bab68c146fd8fb9f4ff5a086e59" is_pre_election: true
I20260926 03:36:35.067024 20663 raft_consensus.cc:2454] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P f10b3a87de534c3da57c237357160083 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2c16ecf9d7b4ee4b23b2a6b1258e75a in term 0.
I20260926 03:36:35.067082 20401 raft_consensus.cc:2454] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P a4f19bab68c146fd8fb9f4ff5a086e59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2c16ecf9d7b4ee4b23b2a6b1258e75a in term 0.
I20260926 03:36:35.067800 20466 leader_election.cc:302] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P e2c16ecf9d7b4ee4b23b2a6b1258e75a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4f19bab68c146fd8fb9f4ff5a086e59, e2c16ecf9d7b4ee4b23b2a6b1258e75a; no voters:
I20260926 03:36:35.068289 20733 raft_consensus.cc:2781] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P e2c16ecf9d7b4ee4b23b2a6b1258e75a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:36:35.068459 20733 raft_consensus.cc:482] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P e2c16ecf9d7b4ee4b23b2a6b1258e75a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:36:35.068590 20733 raft_consensus.cc:3037] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P e2c16ecf9d7b4ee4b23b2a6b1258e75a [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:36:35.071122 20733 raft_consensus.cc:504] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P e2c16ecf9d7b4ee4b23b2a6b1258e75a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c16ecf9d7b4ee4b23b2a6b1258e75a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44391 } } peers { permanent_uuid: "f10b3a87de534c3da57c237357160083" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32893 } } peers { permanent_uuid: "a4f19bab68c146fd8fb9f4ff5a086e59" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41317 } }
I20260926 03:36:35.071790 20733 leader_election.cc:288] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P e2c16ecf9d7b4ee4b23b2a6b1258e75a [CANDIDATE]: Term 1 election: Requested vote from peers f10b3a87de534c3da57c237357160083 (127.0.97.3:32893), a4f19bab68c146fd8fb9f4ff5a086e59 (127.0.97.1:41317)
I20260926 03:36:35.072361 20401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4324adc40dcc42fd8ca8db2c7a5d2ec3" candidate_uuid: "e2c16ecf9d7b4ee4b23b2a6b1258e75a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4f19bab68c146fd8fb9f4ff5a086e59"
I20260926 03:36:35.072359 20663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4324adc40dcc42fd8ca8db2c7a5d2ec3" candidate_uuid: "e2c16ecf9d7b4ee4b23b2a6b1258e75a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f10b3a87de534c3da57c237357160083"
I20260926 03:36:35.072638 20401 raft_consensus.cc:3037] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P a4f19bab68c146fd8fb9f4ff5a086e59 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:36:35.072631 20663 raft_consensus.cc:3037] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P f10b3a87de534c3da57c237357160083 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:36:35.076236 20663 raft_consensus.cc:2454] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P f10b3a87de534c3da57c237357160083 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c16ecf9d7b4ee4b23b2a6b1258e75a in term 1.
I20260926 03:36:35.076249 20401 raft_consensus.cc:2454] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P a4f19bab68c146fd8fb9f4ff5a086e59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c16ecf9d7b4ee4b23b2a6b1258e75a in term 1.
I20260926 03:36:35.076757 20468 leader_election.cc:302] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P e2c16ecf9d7b4ee4b23b2a6b1258e75a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e2c16ecf9d7b4ee4b23b2a6b1258e75a, f10b3a87de534c3da57c237357160083; no voters:
I20260926 03:36:35.077071 20733 raft_consensus.cc:2781] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P e2c16ecf9d7b4ee4b23b2a6b1258e75a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:36:35.077579 20733 raft_consensus.cc:686] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P e2c16ecf9d7b4ee4b23b2a6b1258e75a [term 1 LEADER]: Becoming Leader. State: Replica: e2c16ecf9d7b4ee4b23b2a6b1258e75a, State: Running, Role: LEADER
I20260926 03:36:35.078125 20733 consensus_queue.cc:237] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P e2c16ecf9d7b4ee4b23b2a6b1258e75a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c16ecf9d7b4ee4b23b2a6b1258e75a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44391 } } peers { permanent_uuid: "f10b3a87de534c3da57c237357160083" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32893 } } peers { permanent_uuid: "a4f19bab68c146fd8fb9f4ff5a086e59" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41317 } }
I20260926 03:36:35.084174 20259 catalog_manager.cc:5495] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P e2c16ecf9d7b4ee4b23b2a6b1258e75a reported cstate change: term changed from 0 to 1, leader changed from <none> to e2c16ecf9d7b4ee4b23b2a6b1258e75a (127.0.97.2). New cstate: current_term: 1 leader_uuid: "e2c16ecf9d7b4ee4b23b2a6b1258e75a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c16ecf9d7b4ee4b23b2a6b1258e75a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f10b3a87de534c3da57c237357160083" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4f19bab68c146fd8fb9f4ff5a086e59" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41317 } health_report { overall_health: UNKNOWN } } }
I20260926 03:36:35.153925 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:35.147752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47516 (local address 127.0.97.2:44391)
0926 03:36:35.147953 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:35.147959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:35.148452 (+ 493us) server_negotiation.cc:407] Connection header received
0926 03:36:35.148717 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:35.148722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:35.148829 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:35.148966 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:36:35.149399 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:35.150151 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:35.151652 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:35.152092 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:35.152190 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:35.152714 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:36:35.153383 (+ 669us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:36:35.153509 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:35.153593 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 03:36:35.153679 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 03:36:35.163489 20401 raft_consensus.cc:1261] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P a4f19bab68c146fd8fb9f4ff5a086e59 [term 1 FOLLOWER]: Refusing update from remote peer e2c16ecf9d7b4ee4b23b2a6b1258e75a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:36:35.163496 20663 raft_consensus.cc:1261] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P f10b3a87de534c3da57c237357160083 [term 1 FOLLOWER]: Refusing update from remote peer e2c16ecf9d7b4ee4b23b2a6b1258e75a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:36:35.164548 20738 consensus_queue.cc:1035] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P e2c16ecf9d7b4ee4b23b2a6b1258e75a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f10b3a87de534c3da57c237357160083" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:36:35.165036 20733 consensus_queue.cc:1035] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P e2c16ecf9d7b4ee4b23b2a6b1258e75a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4f19bab68c146fd8fb9f4ff5a086e59" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:36:35.181309 20742 mvcc.cc:205] Tried to move back new op lower bound from 7333452984978554880 to 7333452984644333568. Current Snapshot: MvccSnapshot[applied={T|T < 7333452984978554880}]
I20260926 03:36:35.195881 20258 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:36:35.196234 20258 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:36:35.200423 20258 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db9af3cc94784ab48436e88f16d12084: Sending DeleteTablet for 3 replicas of tablet 4324adc40dcc42fd8ca8db2c7a5d2ec3
I20260926 03:36:35.201804 20643 tablet_service.cc:1514] Processing DeleteTablet for tablet 4324adc40dcc42fd8ca8db2c7a5d2ec3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:36:35 UTC) from {username='slave'} at 127.0.0.1:43836
I20260926 03:36:35.201835 20381 tablet_service.cc:1514] Processing DeleteTablet for tablet 4324adc40dcc42fd8ca8db2c7a5d2ec3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:36:35 UTC) from {username='slave'} at 127.0.0.1:43316
I20260926 03:36:35.202147 20512 tablet_service.cc:1514] Processing DeleteTablet for tablet 4324adc40dcc42fd8ca8db2c7a5d2ec3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:36:35 UTC) from {username='slave'} at 127.0.0.1:47502
I20260926 03:36:35.203984 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 20319
I20260926 03:36:35.213197 20752 tablet_replica.cc:330] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P a4f19bab68c146fd8fb9f4ff5a086e59: stopping tablet replica
I20260926 03:36:35.213526 20754 tablet_replica.cc:330] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P e2c16ecf9d7b4ee4b23b2a6b1258e75a: stopping tablet replica
I20260926 03:36:35.213879 20752 raft_consensus.cc:2229] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P a4f19bab68c146fd8fb9f4ff5a086e59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:36:35.214125 20754 raft_consensus.cc:2229] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P e2c16ecf9d7b4ee4b23b2a6b1258e75a [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:36:35.214583 20752 raft_consensus.cc:2258] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P a4f19bab68c146fd8fb9f4ff5a086e59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:36:35.215060 20754 raft_consensus.cc:2258] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P e2c16ecf9d7b4ee4b23b2a6b1258e75a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:36:35.217690 20754 ts_tablet_manager.cc:1905] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P e2c16ecf9d7b4ee4b23b2a6b1258e75a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:36:35.224290 20752 ts_tablet_manager.cc:1905] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P a4f19bab68c146fd8fb9f4ff5a086e59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:36:35.226467 20753 tablet_replica.cc:330] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P f10b3a87de534c3da57c237357160083: stopping tablet replica
I20260926 03:36:35.227063 20753 raft_consensus.cc:2229] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P f10b3a87de534c3da57c237357160083 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:36:35.227717 20753 raft_consensus.cc:2258] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P f10b3a87de534c3da57c237357160083 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:36:35.231086 20754 ts_tablet_manager.cc:1918] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P e2c16ecf9d7b4ee4b23b2a6b1258e75a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:36:35.231334 20754 log.cc:1196] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P e2c16ecf9d7b4ee4b23b2a6b1258e75a: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-1/wal/wals/4324adc40dcc42fd8ca8db2c7a5d2ec3
I20260926 03:36:35.231979 20754 ts_tablet_manager.cc:1939] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P e2c16ecf9d7b4ee4b23b2a6b1258e75a: Deleting consensus metadata
I20260926 03:36:35.232808 20752 ts_tablet_manager.cc:1918] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P a4f19bab68c146fd8fb9f4ff5a086e59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:36:35.233088 20752 log.cc:1196] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P a4f19bab68c146fd8fb9f4ff5a086e59: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-0/wal/wals/4324adc40dcc42fd8ca8db2c7a5d2ec3
I20260926 03:36:35.233145 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:35.205963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43322 (local address 127.0.97.1:41317)
0926 03:36:35.206274 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:35.206285 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:35.219784 (+ 13499us) server_negotiation.cc:407] Connection header received
0926 03:36:35.220488 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:35.220499 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:35.220681 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:35.220814 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:35.221726 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:35.223288 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:35.228180 (+ 4892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:35.228918 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:35.229028 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:35.230930 (+ 1902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:35.230944 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:35.230965 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:36:35.231949 (+ 984us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:35.232032 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:36:35.232065 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:35.232166 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:35.232450 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 03:36:35.232632 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 03:36:35.233843 20752 ts_tablet_manager.cc:1939] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P a4f19bab68c146fd8fb9f4ff5a086e59: Deleting consensus metadata
I20260926 03:36:35.234684 20441 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:36:35.234875 20244 catalog_manager.cc:4841] TS e2c16ecf9d7b4ee4b23b2a6b1258e75a (127.0.97.2:44391): tablet 4324adc40dcc42fd8ca8db2c7a5d2ec3 (table test-table [id=3a21d31a0ed4421a948b5a60146c4da7]) successfully deleted
I20260926 03:36:35.236908 20753 ts_tablet_manager.cc:1905] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P f10b3a87de534c3da57c237357160083: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:36:35.245064 20753 ts_tablet_manager.cc:1918] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P f10b3a87de534c3da57c237357160083: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:36:35.245285 20753 log.cc:1196] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P f10b3a87de534c3da57c237357160083: Deleting WAL directory at /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790393463219689-388-0/minicluster-data/ts-2/wal/wals/4324adc40dcc42fd8ca8db2c7a5d2ec3
I20260926 03:36:35.245847 20753 ts_tablet_manager.cc:1939] T 4324adc40dcc42fd8ca8db2c7a5d2ec3 P f10b3a87de534c3da57c237357160083: Deleting consensus metadata
I20260926 03:36:35.248215 20246 catalog_manager.cc:4841] TS f10b3a87de534c3da57c237357160083 (127.0.97.3:32893): tablet 4324adc40dcc42fd8ca8db2c7a5d2ec3 (table test-table [id=3a21d31a0ed4421a948b5a60146c4da7]) successfully deleted
I20260926 03:36:35.332921 20244 catalog_manager.cc:4841] TS a4f19bab68c146fd8fb9f4ff5a086e59 (127.0.97.1:41317): tablet 4324adc40dcc42fd8ca8db2c7a5d2ec3 (table test-table [id=3a21d31a0ed4421a948b5a60146c4da7]) successfully deleted
I20260926 03:36:35.333702 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 20319
I20260926 03:36:35.355365 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 20450
I20260926 03:36:35.363319 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:35.356315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47530 (local address 127.0.97.2:44391)
0926 03:36:35.356529 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:35.356536 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:35.356609 (+ 73us) server_negotiation.cc:407] Connection header received
0926 03:36:35.356879 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:35.356883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:35.356991 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:35.357107 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:35.357576 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:35.358486 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:35.359813 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:35.360257 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:35.360391 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:35.361656 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:35.361668 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:35.361687 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:36:35.362474 (+ 787us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:35.362531 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:36:35.362552 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:35.362668 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:35.362811 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 03:36:35.362932 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 03:36:35.364125 20572 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:36:35.490773 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 20450
I20260926 03:36:35.512393 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 20581
I20260926 03:36:35.519738 20724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:35.513401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43838 (local address 127.0.97.3:32893)
0926 03:36:35.513599 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:35.513604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:35.513623 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:36:35.513696 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:35.513700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:35.513816 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:35.513909 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:35.514305 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:35.515231 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:35.516492 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:35.517061 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:35.517174 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:35.518259 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:35.518268 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:35.518286 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:36:35.518984 (+ 698us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:35.519038 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:36:35.519059 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:35.519148 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:35.519251 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 03:36:35.519349 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 03:36:35.520857 20703 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:36:35.616921 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 20581
I20260926 03:36:35.636047 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 20228
I20260926 03:36:35.637470 20289 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:36:35.762370 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 20228
2026-09-26T03:36:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3422 ms)
[----------] 6 tests from EncryptionPolicyTest (24021 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:36:35 dist-test-slave-dt7p krb5kdc[20762](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:36:35 dist-test-slave-dt7p krb5kdc[20762](info): set up 2 sockets
Sep 26 03:36:35 dist-test-slave-dt7p krb5kdc[20762](info): commencing operation
krb5kdc: starting...
W20260926 03:36:36.913427 388 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.004s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:36:37 dist-test-slave-dt7p krb5kdc[20762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:36:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:36:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:36:37.217577 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42199
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:36:37.601291 20775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:36:37.601627 20775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:36:37.601729 20775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:36:37.609364 20775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:36:37.609493 20775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:36:37.609541 20775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:36:37.609601 20775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:36:37.609649 20775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:36:37.620231 20775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42199
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:36:37.621909 20775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:36:37.623540 20775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:36:37.636647 20780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:37.636750 20781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:37.636713 20783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:36:37.638262 20775 server_base.cc:1034] running on GCE node
I20260926 03:36:37.639201 20775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:36:37.640844 20775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:36:37.642053 20775 hybrid_clock.cc:648] HybridClock initialized: now 1790393797642009 us; error 34 us; skew 500 ppm
I20260926 03:36:37.642442 20775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:36:37 dist-test-slave-dt7p krb5kdc[20762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:36:37.645977 20775 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:36:37.647791 20775 webserver.cc:460] Webserver started at http://127.0.0.1:42603/ using document root <none> and password file <none>
I20260926 03:36:37.648548 20775 fs_manager.cc:359] Metadata directory not provided
I20260926 03:36:37.648716 20775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:36:37.649025 20775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:36:37.652974 20775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "3520f4a7d57546448e4935a981ec9854"
format_stamp: "Formatted at 2026-09-26 03:36:37 on dist-test-slave-dt7p"
server_key: "f0ec261ac72ffc76f9ae68fcaf99b92e"
server_key_iv: "c6cc419feecf24e4e9f8961218f75397"
server_key_version: "encryptionkey@0"
I20260926 03:36:37.653810 20775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "3520f4a7d57546448e4935a981ec9854"
format_stamp: "Formatted at 2026-09-26 03:36:37 on dist-test-slave-dt7p"
server_key: "f0ec261ac72ffc76f9ae68fcaf99b92e"
server_key_iv: "c6cc419feecf24e4e9f8961218f75397"
server_key_version: "encryptionkey@0"
I20260926 03:36:37.659783 20775 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260926 03:36:37.663784 20790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:37.665438 20775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:36:37.665640 20775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "3520f4a7d57546448e4935a981ec9854"
format_stamp: "Formatted at 2026-09-26 03:36:37 on dist-test-slave-dt7p"
server_key: "f0ec261ac72ffc76f9ae68fcaf99b92e"
server_key_iv: "c6cc419feecf24e4e9f8961218f75397"
server_key_version: "encryptionkey@0"
I20260926 03:36:37.665879 20775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:36:37.698146 20775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:36:37.704604 20775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:36:37.705034 20775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:36:37.726953 20775 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42199
I20260926 03:36:37.726976 20841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42199 every 8 connection(s)
I20260926 03:36:37.728718 20775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:36:37.733193 20842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:36:37.738118 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 20775
I20260926 03:36:37.738375 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:36:37.739058 388 external_mini_cluster.cc:1428] Setting key dac60c30ed05d65cd38442d685b39304
I20260926 03:36:37.746475 20842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3520f4a7d57546448e4935a981ec9854: Bootstrap starting.
I20260926 03:36:37.750707 20842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3520f4a7d57546448e4935a981ec9854: Neither blocks nor log segments found. Creating new log.
I20260926 03:36:37.752204 20842 log.cc:824] T 00000000000000000000000000000000 P 3520f4a7d57546448e4935a981ec9854: Log is configured to *not* fsync() on all Append() calls
I20260926 03:36:37.755355 20842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3520f4a7d57546448e4935a981ec9854: No bootstrap required, opened a new log
Sep 26 03:36:37 dist-test-slave-dt7p krb5kdc[20762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 03:36:37.765928 20845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:37.741261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51895 (local address 127.0.0.1:42199)
0926 03:36:37.742156 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:37.742175 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:37.742250 (+ 75us) server_negotiation.cc:407] Connection header received
0926 03:36:37.744907 (+ 2657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:37.744941 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:37.745493 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:37.745959 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:37.747405 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:37.749611 (+ 2206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:37.751048 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:37.752201 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:37.752379 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:37.757034 (+ 4655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:37.757044 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:37.757070 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:36:37.757132 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:37.761483 (+ 4351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:37.762285 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:37.762295 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:37.762308 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:37.762535 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:37.762965 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:37.762970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:37.762974 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:37.763700 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:37.764064 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:37.764412 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 03:36:37.764774 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":361,"threads_started":1}
I20260926 03:36:37.766436 20842 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3520f4a7d57546448e4935a981ec9854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3520f4a7d57546448e4935a981ec9854" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42199 } }
I20260926 03:36:37.766844 20842 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3520f4a7d57546448e4935a981ec9854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:36:37.766958 20842 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3520f4a7d57546448e4935a981ec9854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3520f4a7d57546448e4935a981ec9854, State: Initialized, Role: FOLLOWER
I20260926 03:36:37.767624 20842 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3520f4a7d57546448e4935a981ec9854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3520f4a7d57546448e4935a981ec9854" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42199 } }
I20260926 03:36:37.767903 20842 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3520f4a7d57546448e4935a981ec9854 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:36:37.768055 20842 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3520f4a7d57546448e4935a981ec9854 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:36:37.768244 20842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3520f4a7d57546448e4935a981ec9854 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:36:37.770363 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:36:37.771057 20842 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3520f4a7d57546448e4935a981ec9854 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3520f4a7d57546448e4935a981ec9854" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42199 } }
I20260926 03:36:37.771611 20842 leader_election.cc:302] T 00000000000000000000000000000000 P 3520f4a7d57546448e4935a981ec9854 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3520f4a7d57546448e4935a981ec9854; no voters:
I20260926 03:36:37.772244 20842 leader_election.cc:288] T 00000000000000000000000000000000 P 3520f4a7d57546448e4935a981ec9854 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:36:37.772409 20847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3520f4a7d57546448e4935a981ec9854 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:36:37.773116 20847 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3520f4a7d57546448e4935a981ec9854 [term 1 LEADER]: Becoming Leader. State: Replica: 3520f4a7d57546448e4935a981ec9854, State: Running, Role: LEADER
I20260926 03:36:37.773828 20847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3520f4a7d57546448e4935a981ec9854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3520f4a7d57546448e4935a981ec9854" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42199 } }
I20260926 03:36:37.777740 20849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3520f4a7d57546448e4935a981ec9854 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3520f4a7d57546448e4935a981ec9854" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3520f4a7d57546448e4935a981ec9854" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42199 } } }
I20260926 03:36:37.778331 20849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3520f4a7d57546448e4935a981ec9854 [sys.catalog]: This master's current role is: LEADER
I20260926 03:36:37.779903 20849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3520f4a7d57546448e4935a981ec9854 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3520f4a7d57546448e4935a981ec9854. Latest consensus state: current_term: 1 leader_uuid: "3520f4a7d57546448e4935a981ec9854" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3520f4a7d57546448e4935a981ec9854" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42199 } } }
I20260926 03:36:37.780262 20849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3520f4a7d57546448e4935a981ec9854 [sys.catalog]: This master's current role is: LEADER
I20260926 03:36:37.786973 20842 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3520f4a7d57546448e4935a981ec9854 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:36:37.787245 20853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:36:37.793282 20853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:36:37.807152 20853 catalog_manager.cc:1347] Generated new cluster ID: 6868d04f187a4c3693ed5839b4d7879f
I20260926 03:36:37.807286 20853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:36:37.808902 20845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:37.786054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55225 (local address 127.0.0.1:42199)
0926 03:36:37.786229 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:37.786236 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:37.786378 (+ 142us) server_negotiation.cc:407] Connection header received
0926 03:36:37.786620 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:37.786627 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:37.786717 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:37.786824 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:37.789001 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:37.790676 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:37.793336 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:37.794314 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:37.794444 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:37.798952 (+ 4508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:37.798962 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:37.798970 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:36:37.799025 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:37.804062 (+ 5037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:37.805661 (+ 1599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:37.805670 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:37.805676 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:37.805869 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:37.806598 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:37.806607 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:37.806613 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:37.807184 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:37.807496 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:37.808276 (+ 780us) server_negotiation.cc:299] Negotiation successful
0926 03:36:37.808475 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 03:36:37.848237 20853 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:36:37.850317 20853 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:36:37.859869 20853 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3520f4a7d57546448e4935a981ec9854: Generated new TSK 0
I20260926 03:36:37.860895 20853 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:36:37.877086 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 20775
I20260926 03:36:37.878816 20837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:36:37.989663 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 20775
W20260926 03:36:38.010205 20760 connection.cc:535] client connection to 127.0.0.1:42199 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42199): Connection reset by peer (error 104)
2026-09-26T03:36:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2221 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:36:38 dist-test-slave-dt7p krb5kdc[20875](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:36:38 dist-test-slave-dt7p krb5kdc[20875](info): set up 2 sockets
Sep 26 03:36:38 dist-test-slave-dt7p krb5kdc[20875](info): commencing operation
krb5kdc: starting...
W20260926 03:36:39.153425 388 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.004s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:36:39 dist-test-slave-dt7p krb5kdc[20875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:36:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:36:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:36:39.441741 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42213
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:36:39.825759 20888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:36:39.826094 20888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:36:39.826274 20888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:36:39.833742 20888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:36:39.833850 20888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:36:39.833961 20888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:36:39.834070 20888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:36:39.834152 20888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:36:39.844341 20888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42213
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:36:39.845953 20888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:36:39.847572 20888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:36:39.860946 20896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:39.861019 20893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:39.860989 20894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:36:39.862288 20888 server_base.cc:1034] running on GCE node
I20260926 03:36:39.863106 20888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:36:39.864729 20888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:36:39.865978 20888 hybrid_clock.cc:648] HybridClock initialized: now 1790393799865926 us; error 43 us; skew 500 ppm
I20260926 03:36:39.866371 20888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:36:39 dist-test-slave-dt7p krb5kdc[20875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393799, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:36:39.869861 20888 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:36:39.871562 20888 webserver.cc:460] Webserver started at http://127.0.0.1:39417/ using document root <none> and password file <none>
I20260926 03:36:39.872339 20888 fs_manager.cc:359] Metadata directory not provided
I20260926 03:36:39.872488 20888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:36:39.872869 20888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:36:39.876189 20888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "33b55caa2b494cc282d3c3eee2907cde"
format_stamp: "Formatted at 2026-09-26 03:36:39 on dist-test-slave-dt7p"
server_key: "ea2a3a48593680894abce90f0bf04c55"
server_key_iv: "43017d436f7adc59b041a3fbc9e8156a"
server_key_version: "encryptionkey@0"
I20260926 03:36:39.877134 20888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "33b55caa2b494cc282d3c3eee2907cde"
format_stamp: "Formatted at 2026-09-26 03:36:39 on dist-test-slave-dt7p"
server_key: "ea2a3a48593680894abce90f0bf04c55"
server_key_iv: "43017d436f7adc59b041a3fbc9e8156a"
server_key_version: "encryptionkey@0"
I20260926 03:36:39.882836 20888 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:36:39.886672 20903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:39.888108 20888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:36:39.888303 20888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "33b55caa2b494cc282d3c3eee2907cde"
format_stamp: "Formatted at 2026-09-26 03:36:39 on dist-test-slave-dt7p"
server_key: "ea2a3a48593680894abce90f0bf04c55"
server_key_iv: "43017d436f7adc59b041a3fbc9e8156a"
server_key_version: "encryptionkey@0"
I20260926 03:36:39.888521 20888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:36:39.909255 20888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:36:39.913947 20888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:36:39.914278 20888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:36:39.935372 20888 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42213
I20260926 03:36:39.935395 20954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42213 every 8 connection(s)
I20260926 03:36:39.937129 20888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:36:39.941457 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 20888
I20260926 03:36:39.941679 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:36:39.941731 20955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:36:39.942310 388 external_mini_cluster.cc:1428] Setting key c0001062731caaa36096c32521da667f
I20260926 03:36:39.958280 20958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:39.944370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51929 (local address 127.0.0.1:42213)
0926 03:36:39.945361 (+ 991us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:39.945387 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:39.945470 (+ 83us) server_negotiation.cc:407] Connection header received
0926 03:36:39.948128 (+ 2658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:39.948157 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:39.948812 (+ 655us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:39.949218 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:39.949885 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:39.951339 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:39.953302 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:39.954031 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:39.954165 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:39.955691 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:39.955701 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:39.955717 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:36:39.956376 (+ 659us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:39.956460 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:36:39.956486 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:39.956670 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:39.956935 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 03:36:39.957165 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":713,"thread_start_us":451,"threads_started":1}
I20260926 03:36:39.958662 20955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33b55caa2b494cc282d3c3eee2907cde: Bootstrap starting.
I20260926 03:36:39.962787 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:36:39.962960 20955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33b55caa2b494cc282d3c3eee2907cde: Neither blocks nor log segments found. Creating new log.
I20260926 03:36:39.964283 20955 log.cc:824] T 00000000000000000000000000000000 P 33b55caa2b494cc282d3c3eee2907cde: Log is configured to *not* fsync() on all Append() calls
I20260926 03:36:39.968055 20955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33b55caa2b494cc282d3c3eee2907cde: No bootstrap required, opened a new log
I20260926 03:36:39.976027 20958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:39.968718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47983 (local address 127.0.0.1:42213)
0926 03:36:39.968933 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:39.968940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:39.969376 (+ 436us) server_negotiation.cc:407] Connection header received
0926 03:36:39.969537 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:39.969543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:39.969634 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:39.969747 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:39.970247 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:39.971531 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:39.973080 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:39.973840 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:39.973996 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:39.975160 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:39.975166 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:39.975171 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:36:39.975194 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:39.975259 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:36:39.975290 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:39.975405 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:39.975585 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 03:36:39.975655 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 03:36:39.978782 20955 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33b55caa2b494cc282d3c3eee2907cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b55caa2b494cc282d3c3eee2907cde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42213 } }
I20260926 03:36:39.979233 20955 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33b55caa2b494cc282d3c3eee2907cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:36:39.979368 20955 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33b55caa2b494cc282d3c3eee2907cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33b55caa2b494cc282d3c3eee2907cde, State: Initialized, Role: FOLLOWER
I20260926 03:36:39.980020 20955 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33b55caa2b494cc282d3c3eee2907cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b55caa2b494cc282d3c3eee2907cde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42213 } }
I20260926 03:36:39.980290 20955 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33b55caa2b494cc282d3c3eee2907cde [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:36:39.980461 20955 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33b55caa2b494cc282d3c3eee2907cde [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:36:39.980655 20955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33b55caa2b494cc282d3c3eee2907cde [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:36:39.983181 20955 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33b55caa2b494cc282d3c3eee2907cde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b55caa2b494cc282d3c3eee2907cde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42213 } }
I20260926 03:36:39.983753 20955 leader_election.cc:302] T 00000000000000000000000000000000 P 33b55caa2b494cc282d3c3eee2907cde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33b55caa2b494cc282d3c3eee2907cde; no voters:
I20260926 03:36:39.984377 20955 leader_election.cc:288] T 00000000000000000000000000000000 P 33b55caa2b494cc282d3c3eee2907cde [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:36:39.984508 20965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33b55caa2b494cc282d3c3eee2907cde [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:36:39.985438 20965 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33b55caa2b494cc282d3c3eee2907cde [term 1 LEADER]: Becoming Leader. State: Replica: 33b55caa2b494cc282d3c3eee2907cde, State: Running, Role: LEADER
I20260926 03:36:39.986078 20965 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33b55caa2b494cc282d3c3eee2907cde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b55caa2b494cc282d3c3eee2907cde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42213 } }
I20260926 03:36:39.986908 20955 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33b55caa2b494cc282d3c3eee2907cde [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:36:39.989152 20966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33b55caa2b494cc282d3c3eee2907cde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33b55caa2b494cc282d3c3eee2907cde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b55caa2b494cc282d3c3eee2907cde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42213 } } }
I20260926 03:36:39.989534 20966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33b55caa2b494cc282d3c3eee2907cde [sys.catalog]: This master's current role is: LEADER
I20260926 03:36:39.989991 20967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33b55caa2b494cc282d3c3eee2907cde [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33b55caa2b494cc282d3c3eee2907cde. Latest consensus state: current_term: 1 leader_uuid: "33b55caa2b494cc282d3c3eee2907cde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b55caa2b494cc282d3c3eee2907cde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42213 } } }
I20260926 03:36:39.990295 20967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33b55caa2b494cc282d3c3eee2907cde [sys.catalog]: This master's current role is: LEADER
I20260926 03:36:39.992267 20971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:36:39.998464 20971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:36:40.009310 20971 catalog_manager.cc:1347] Generated new cluster ID: d648202c2cd04bbb8085f89c8b756322
I20260926 03:36:40.009461 20971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:36:40.029435 20971 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:36:40.031122 20971 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:36:40.041648 20971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33b55caa2b494cc282d3c3eee2907cde: Generated new TSK 0
I20260926 03:36:40.042893 20971 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:36:40.054980 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 20888
W20260926 03:36:40.056322 20950 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51929
W20260926 03:36:40.057384 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 20888: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:36:40.057586 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 20888
W20260926 03:36:40.077807 20873 connection.cc:535] client connection to 127.0.0.1:42213 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42213): Connection reset by peer (error 104)
2026-09-26T03:36:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2067 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:36:40 dist-test-slave-dt7p krb5kdc[20986](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:36:40 dist-test-slave-dt7p krb5kdc[20986](info): set up 2 sockets
Sep 26 03:36:40 dist-test-slave-dt7p krb5kdc[20986](info): commencing operation
krb5kdc: starting...
W20260926 03:36:41.193336 388 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.003s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:36:41 dist-test-slave-dt7p krb5kdc[20986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:36:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:36:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:36:41.470000 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44475
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:36:41.847414 20999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:36:41.847810 20999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:36:41.847990 20999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:36:41.855473 20999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:36:41.855636 20999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:36:41.855749 20999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:36:41.855870 20999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:36:41.855958 20999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:36:41.866291 20999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44475
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:36:41.867872 20999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:36:41.869427 20999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:36:41.882545 21007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:41.882645 21005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:41.883180 21004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:36:41.883808 20999 server_base.cc:1034] running on GCE node
I20260926 03:36:41.884727 20999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:36:41.886408 20999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:36:41.887653 20999 hybrid_clock.cc:648] HybridClock initialized: now 1790393801887582 us; error 51 us; skew 500 ppm
I20260926 03:36:41.888037 20999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:36:41 dist-test-slave-dt7p krb5kdc[20986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:36:41.891561 20999 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:36:41.893180 20999 webserver.cc:460] Webserver started at http://127.0.0.1:42851/ using document root <none> and password file <none>
I20260926 03:36:41.893941 20999 fs_manager.cc:359] Metadata directory not provided
I20260926 03:36:41.894107 20999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:36:41.894452 20999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:36:41.898025 20999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "f30e16fd33f445748d8a96699637be5a"
format_stamp: "Formatted at 2026-09-26 03:36:41 on dist-test-slave-dt7p"
server_key: "87c6227878cb9fd1215d96fefd46f126"
server_key_iv: "8e49e377bf367156235da7d20a979876"
server_key_version: "encryptionkey@0"
I20260926 03:36:41.898867 20999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "f30e16fd33f445748d8a96699637be5a"
format_stamp: "Formatted at 2026-09-26 03:36:41 on dist-test-slave-dt7p"
server_key: "87c6227878cb9fd1215d96fefd46f126"
server_key_iv: "8e49e377bf367156235da7d20a979876"
server_key_version: "encryptionkey@0"
I20260926 03:36:41.906908 20999 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.002s sys 0.007s
I20260926 03:36:41.911343 21014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:41.912814 20999 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:36:41.913002 20999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "f30e16fd33f445748d8a96699637be5a"
format_stamp: "Formatted at 2026-09-26 03:36:41 on dist-test-slave-dt7p"
server_key: "87c6227878cb9fd1215d96fefd46f126"
server_key_iv: "8e49e377bf367156235da7d20a979876"
server_key_version: "encryptionkey@0"
I20260926 03:36:41.913220 20999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:36:41.919230 20999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:36:41.923817 20999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:36:41.924118 20999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:36:41.945206 20999 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44475
I20260926 03:36:41.945233 21065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44475 every 8 connection(s)
I20260926 03:36:41.947012 20999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:36:41.947784 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 20999
I20260926 03:36:41.948017 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:36:41.948691 388 external_mini_cluster.cc:1428] Setting key adec085252e1b5fb0b77bcd4d76cdb0c
I20260926 03:36:41.952450 21066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:36:41.959522 21069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:41.950813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40423 (local address 127.0.0.1:44475)
0926 03:36:41.951650 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:41.951663 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:41.951711 (+ 48us) server_negotiation.cc:407] Connection header received
0926 03:36:41.954374 (+ 2663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:41.954408 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:41.954957 (+ 549us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:41.956031 (+ 1074us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:41.957579 (+ 1548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:41.957585 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:41.957599 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:36:41.957613 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:41.957697 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:36:41.957718 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:41.957990 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:41.958142 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 03:36:41.958420 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":338,"threads_started":1}
I20260926 03:36:41.963842 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:36:41.966850 21066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f30e16fd33f445748d8a96699637be5a: Bootstrap starting.
I20260926 03:36:41.971339 21066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f30e16fd33f445748d8a96699637be5a: Neither blocks nor log segments found. Creating new log.
I20260926 03:36:41.972226 21069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:41.969649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60163 (local address 127.0.0.1:44475)
0926 03:36:41.969809 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:41.969813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:41.970159 (+ 346us) server_negotiation.cc:407] Connection header received
0926 03:36:41.970356 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:41.970360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:41.970412 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:41.970489 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:41.971508 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:41.971515 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:41.971520 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:36:41.971543 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:41.971588 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:36:41.971603 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:41.971695 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:41.971860 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 03:36:41.971908 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 03:36:41.972836 21066 log.cc:824] T 00000000000000000000000000000000 P f30e16fd33f445748d8a96699637be5a: Log is configured to *not* fsync() on all Append() calls
I20260926 03:36:41.975818 21066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f30e16fd33f445748d8a96699637be5a: No bootstrap required, opened a new log
I20260926 03:36:41.983625 21066 raft_consensus.cc:348] T 00000000000000000000000000000000 P f30e16fd33f445748d8a96699637be5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30e16fd33f445748d8a96699637be5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44475 } }
I20260926 03:36:41.984017 21066 raft_consensus.cc:374] T 00000000000000000000000000000000 P f30e16fd33f445748d8a96699637be5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:36:41.984097 21066 raft_consensus.cc:729] T 00000000000000000000000000000000 P f30e16fd33f445748d8a96699637be5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f30e16fd33f445748d8a96699637be5a, State: Initialized, Role: FOLLOWER
I20260926 03:36:41.984700 21066 consensus_queue.cc:260] T 00000000000000000000000000000000 P f30e16fd33f445748d8a96699637be5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30e16fd33f445748d8a96699637be5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44475 } }
I20260926 03:36:41.985040 21066 raft_consensus.cc:388] T 00000000000000000000000000000000 P f30e16fd33f445748d8a96699637be5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:36:41.985211 21066 raft_consensus.cc:482] T 00000000000000000000000000000000 P f30e16fd33f445748d8a96699637be5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:36:41.985426 21066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f30e16fd33f445748d8a96699637be5a [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:36:41.987776 21066 raft_consensus.cc:504] T 00000000000000000000000000000000 P f30e16fd33f445748d8a96699637be5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30e16fd33f445748d8a96699637be5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44475 } }
I20260926 03:36:41.988372 21066 leader_election.cc:302] T 00000000000000000000000000000000 P f30e16fd33f445748d8a96699637be5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f30e16fd33f445748d8a96699637be5a; no voters:
I20260926 03:36:41.989171 21066 leader_election.cc:288] T 00000000000000000000000000000000 P f30e16fd33f445748d8a96699637be5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:36:41.989332 21076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f30e16fd33f445748d8a96699637be5a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:36:41.990316 21076 raft_consensus.cc:686] T 00000000000000000000000000000000 P f30e16fd33f445748d8a96699637be5a [term 1 LEADER]: Becoming Leader. State: Replica: f30e16fd33f445748d8a96699637be5a, State: Running, Role: LEADER
I20260926 03:36:41.990931 21076 consensus_queue.cc:237] T 00000000000000000000000000000000 P f30e16fd33f445748d8a96699637be5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30e16fd33f445748d8a96699637be5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44475 } }
I20260926 03:36:41.991576 21066 sys_catalog.cc:564] T 00000000000000000000000000000000 P f30e16fd33f445748d8a96699637be5a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:36:41.993947 21077 sys_catalog.cc:455] T 00000000000000000000000000000000 P f30e16fd33f445748d8a96699637be5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f30e16fd33f445748d8a96699637be5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30e16fd33f445748d8a96699637be5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44475 } } }
I20260926 03:36:41.994258 21078 sys_catalog.cc:455] T 00000000000000000000000000000000 P f30e16fd33f445748d8a96699637be5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f30e16fd33f445748d8a96699637be5a. Latest consensus state: current_term: 1 leader_uuid: "f30e16fd33f445748d8a96699637be5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30e16fd33f445748d8a96699637be5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44475 } } }
I20260926 03:36:41.994549 21077 sys_catalog.cc:458] T 00000000000000000000000000000000 P f30e16fd33f445748d8a96699637be5a [sys.catalog]: This master's current role is: LEADER
I20260926 03:36:41.994585 21078 sys_catalog.cc:458] T 00000000000000000000000000000000 P f30e16fd33f445748d8a96699637be5a [sys.catalog]: This master's current role is: LEADER
I20260926 03:36:41.996836 21082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:36:42.003934 21082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:36:42.014779 21082 catalog_manager.cc:1347] Generated new cluster ID: 3455bf892df14a46966edea349b53325
I20260926 03:36:42.014925 21082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:36:42.037815 21082 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:36:42.038506 21082 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:36:42.055397 21082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f30e16fd33f445748d8a96699637be5a: Generated new TSK 0
I20260926 03:36:42.056797 21082 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:36:42.067319 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 20999
W20260926 03:36:42.068509 21061 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40423
W20260926 03:36:42.069294 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 20999: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:36:42.069440 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 20999
2026-09-26T03:36:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2011 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:36:42 dist-test-slave-dt7p krb5kdc[21097](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:36:42 dist-test-slave-dt7p krb5kdc[21097](info): set up 2 sockets
Sep 26 03:36:42 dist-test-slave-dt7p krb5kdc[21097](info): commencing operation
krb5kdc: starting...
W20260926 03:36:43.249239 388 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:36:43 dist-test-slave-dt7p krb5kdc[21097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:36:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:36:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:36:43.552202 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43741
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:36:43.933357 21110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:36:43.933729 21110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:36:43.933905 21110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:36:43.941502 21110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:36:43.941654 21110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:36:43.941761 21110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:36:43.941874 21110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:36:43.941951 21110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:36:43.952060 21110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:36:43.953649 21110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:36:43.955175 21110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:36:43.970827 21115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:43.972072 21116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:43.972518 21118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:36:43.974195 21110 server_base.cc:1034] running on GCE node
I20260926 03:36:43.975311 21110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:36:43.977548 21110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:36:43.978835 21110 hybrid_clock.cc:648] HybridClock initialized: now 1790393803978775 us; error 43 us; skew 500 ppm
I20260926 03:36:43.979321 21110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:36:43 dist-test-slave-dt7p krb5kdc[21097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:36:43.984051 21110 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:36:43.986225 21110 webserver.cc:460] Webserver started at http://127.0.0.1:35581/ using document root <none> and password file <none>
I20260926 03:36:43.987351 21110 fs_manager.cc:359] Metadata directory not provided
I20260926 03:36:43.987500 21110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:36:43.987905 21110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:36:43.992914 21110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "59e76dd2f0cf4053a85215e74e7bece6"
format_stamp: "Formatted at 2026-09-26 03:36:43 on dist-test-slave-dt7p"
server_key: "04cbc2c66c055d3fec89fe312e764301"
server_key_iv: "158e8c47bcc8ea701b6a99113c4897fd"
server_key_version: "encryptionkey@0"
I20260926 03:36:43.994068 21110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "59e76dd2f0cf4053a85215e74e7bece6"
format_stamp: "Formatted at 2026-09-26 03:36:43 on dist-test-slave-dt7p"
server_key: "04cbc2c66c055d3fec89fe312e764301"
server_key_iv: "158e8c47bcc8ea701b6a99113c4897fd"
server_key_version: "encryptionkey@0"
I20260926 03:36:44.002311 21110 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s
I20260926 03:36:44.006647 21125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:44.008550 21110 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260926 03:36:44.008769 21110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "59e76dd2f0cf4053a85215e74e7bece6"
format_stamp: "Formatted at 2026-09-26 03:36:43 on dist-test-slave-dt7p"
server_key: "04cbc2c66c055d3fec89fe312e764301"
server_key_iv: "158e8c47bcc8ea701b6a99113c4897fd"
server_key_version: "encryptionkey@0"
I20260926 03:36:44.009029 21110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:36:44.033970 21110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:36:44.038661 21110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:36:44.038985 21110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:36:44.060636 21110 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43741
I20260926 03:36:44.060693 21176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43741 every 8 connection(s)
I20260926 03:36:44.062291 21110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:36:44.066776 21177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:36:44.071751 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 21110
I20260926 03:36:44.072005 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:36:44.072603 388 external_mini_cluster.cc:1428] Setting key 2ee1e8ec462f7715c6a3d41b045c692b
I20260926 03:36:44.079116 21177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59e76dd2f0cf4053a85215e74e7bece6: Bootstrap starting.
I20260926 03:36:44.083174 21177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59e76dd2f0cf4053a85215e74e7bece6: Neither blocks nor log segments found. Creating new log.
I20260926 03:36:44.084471 21177 log.cc:824] T 00000000000000000000000000000000 P 59e76dd2f0cf4053a85215e74e7bece6: Log is configured to *not* fsync() on all Append() calls
I20260926 03:36:44.088195 21177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59e76dd2f0cf4053a85215e74e7bece6: No bootstrap required, opened a new log
Sep 26 03:36:44 dist-test-slave-dt7p krb5kdc[21097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 03:36:44.099438 21177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59e76dd2f0cf4053a85215e74e7bece6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e76dd2f0cf4053a85215e74e7bece6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43741 } }
I20260926 03:36:44.099970 21177 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59e76dd2f0cf4053a85215e74e7bece6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:36:44.100021 21180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:44.074834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55659 (local address 127.0.0.1:43741)
0926 03:36:44.075841 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:44.075854 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:44.075902 (+ 48us) server_negotiation.cc:407] Connection header received
0926 03:36:44.077981 (+ 2079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:44.078006 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:44.078445 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:44.078808 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:44.080749 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:44.082262 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:44.084144 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:44.084951 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:44.085217 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:44.090385 (+ 5168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:44.090398 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:44.090423 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:36:44.090469 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:44.094873 (+ 4404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:44.095656 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:44.095665 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:44.095675 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:44.095911 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:44.096370 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:44.096375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:44.096378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:44.097206 (+ 828us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:44.097558 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:44.097963 (+ 405us) server_negotiation.cc:299] Negotiation successful
0926 03:36:44.098482 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":734,"thread_start_us":490,"threads_started":1}
I20260926 03:36:44.100119 21177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59e76dd2f0cf4053a85215e74e7bece6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59e76dd2f0cf4053a85215e74e7bece6, State: Initialized, Role: FOLLOWER
I20260926 03:36:44.100873 21177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59e76dd2f0cf4053a85215e74e7bece6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e76dd2f0cf4053a85215e74e7bece6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43741 } }
I20260926 03:36:44.101151 21177 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59e76dd2f0cf4053a85215e74e7bece6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:36:44.101313 21177 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59e76dd2f0cf4053a85215e74e7bece6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:36:44.101514 21177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59e76dd2f0cf4053a85215e74e7bece6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:36:44.104277 21177 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59e76dd2f0cf4053a85215e74e7bece6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e76dd2f0cf4053a85215e74e7bece6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43741 } }
I20260926 03:36:44.104482 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:36:44.104969 21177 leader_election.cc:302] T 00000000000000000000000000000000 P 59e76dd2f0cf4053a85215e74e7bece6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59e76dd2f0cf4053a85215e74e7bece6; no voters:
I20260926 03:36:44.105669 21177 leader_election.cc:288] T 00000000000000000000000000000000 P 59e76dd2f0cf4053a85215e74e7bece6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:36:44.105883 21182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59e76dd2f0cf4053a85215e74e7bece6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:36:44.107993 21177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59e76dd2f0cf4053a85215e74e7bece6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:36:44.108906 21182 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59e76dd2f0cf4053a85215e74e7bece6 [term 1 LEADER]: Becoming Leader. State: Replica: 59e76dd2f0cf4053a85215e74e7bece6, State: Running, Role: LEADER
I20260926 03:36:44.109907 21182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59e76dd2f0cf4053a85215e74e7bece6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e76dd2f0cf4053a85215e74e7bece6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43741 } }
I20260926 03:36:44.113790 21184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59e76dd2f0cf4053a85215e74e7bece6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59e76dd2f0cf4053a85215e74e7bece6. Latest consensus state: current_term: 1 leader_uuid: "59e76dd2f0cf4053a85215e74e7bece6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e76dd2f0cf4053a85215e74e7bece6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43741 } } }
I20260926 03:36:44.114279 21184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59e76dd2f0cf4053a85215e74e7bece6 [sys.catalog]: This master's current role is: LEADER
I20260926 03:36:44.114966 21183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59e76dd2f0cf4053a85215e74e7bece6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59e76dd2f0cf4053a85215e74e7bece6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59e76dd2f0cf4053a85215e74e7bece6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43741 } } }
I20260926 03:36:44.115361 21183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59e76dd2f0cf4053a85215e74e7bece6 [sys.catalog]: This master's current role is: LEADER
I20260926 03:36:44.118741 21191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:36:44.125438 21191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:36:44.139546 21191 catalog_manager.cc:1347] Generated new cluster ID: acb6dd02be084f7aa37129671851affa
I20260926 03:36:44.139681 21191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:36:44.142234 21180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:44.119749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60391 (local address 127.0.0.1:43741)
0926 03:36:44.119961 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:44.119968 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:44.120489 (+ 521us) server_negotiation.cc:407] Connection header received
0926 03:36:44.120794 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:44.120803 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:44.120930 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:44.121037 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:44.123473 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:44.124703 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:44.126856 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:44.127661 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:44.127820 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:44.134708 (+ 6888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:44.134716 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:44.134722 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:36:44.134766 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:44.139003 (+ 4237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:44.139932 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:44.139938 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:44.139941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:44.140098 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:44.140516 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:44.140522 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:44.140526 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:44.140932 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:44.141209 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:44.141584 (+ 375us) server_negotiation.cc:299] Negotiation successful
0926 03:36:44.141837 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 03:36:44.179206 21191 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:36:44.180760 21191 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:36:44.189214 21191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59e76dd2f0cf4053a85215e74e7bece6: Generated new TSK 0
I20260926 03:36:44.190429 21191 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:36:44.202458 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 21110
I20260926 03:36:44.203809 21172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:36:44.305001 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 21110
2026-09-26T03:36:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2235 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:36:44 dist-test-slave-dt7p krb5kdc[21209](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:36:44 dist-test-slave-dt7p krb5kdc[21209](info): set up 2 sockets
Sep 26 03:36:44 dist-test-slave-dt7p krb5kdc[21209](info): commencing operation
krb5kdc: starting...
W20260926 03:36:45.433398 388 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:36:45 dist-test-slave-dt7p krb5kdc[21209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:36:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:36:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:36:45.732249 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35195
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:36:46.122169 21222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:36:46.122488 21222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:36:46.122586 21222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:36:46.129968 21222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:36:46.130081 21222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:36:46.130129 21222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:36:46.130194 21222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:36:46.130234 21222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:36:46.140239 21222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:36:46.141690 21222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:36:46.143216 21222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:36:46.156751 21230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:46.156805 21228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:46.156821 21227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:36:46.157992 21222 server_base.cc:1034] running on GCE node
I20260926 03:36:46.158813 21222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:36:46.160317 21222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:36:46.161511 21222 hybrid_clock.cc:648] HybridClock initialized: now 1790393806161442 us; error 57 us; skew 500 ppm
I20260926 03:36:46.161892 21222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:36:46 dist-test-slave-dt7p krb5kdc[21209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:36:46.165380 21222 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:36:46.166893 21222 webserver.cc:460] Webserver started at http://127.0.0.1:33081/ using document root <none> and password file <none>
I20260926 03:36:46.167654 21222 fs_manager.cc:359] Metadata directory not provided
I20260926 03:36:46.167789 21222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:36:46.168139 21222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:36:46.171820 21222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "9103752b4bdc4023bc1ccf23d819db73"
format_stamp: "Formatted at 2026-09-26 03:36:46 on dist-test-slave-dt7p"
server_key: "b4144c6a50b20b98196493093364fe3b"
server_key_iv: "0fb755679309793c56ddec35c8ef9119"
server_key_version: "encryptionkey@0"
I20260926 03:36:46.172816 21222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "9103752b4bdc4023bc1ccf23d819db73"
format_stamp: "Formatted at 2026-09-26 03:36:46 on dist-test-slave-dt7p"
server_key: "b4144c6a50b20b98196493093364fe3b"
server_key_iv: "0fb755679309793c56ddec35c8ef9119"
server_key_version: "encryptionkey@0"
I20260926 03:36:46.178637 21222 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:36:46.182497 21237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:46.183921 21222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:36:46.184146 21222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "9103752b4bdc4023bc1ccf23d819db73"
format_stamp: "Formatted at 2026-09-26 03:36:46 on dist-test-slave-dt7p"
server_key: "b4144c6a50b20b98196493093364fe3b"
server_key_iv: "0fb755679309793c56ddec35c8ef9119"
server_key_version: "encryptionkey@0"
I20260926 03:36:46.184370 21222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:36:46.205848 21222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:36:46.210253 21222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:36:46.210682 21222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:36:46.232030 21222 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35195
I20260926 03:36:46.232055 21288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35195 every 8 connection(s)
I20260926 03:36:46.233695 21222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:36:46.238094 21289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:36:46.242970 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 21222
I20260926 03:36:46.243247 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:36:46.243762 388 external_mini_cluster.cc:1428] Setting key 9e3e66407a9821b2334eb923194ed411
I20260926 03:36:46.250871 21289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9103752b4bdc4023bc1ccf23d819db73: Bootstrap starting.
I20260926 03:36:46.254916 21289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9103752b4bdc4023bc1ccf23d819db73: Neither blocks nor log segments found. Creating new log.
I20260926 03:36:46.256377 21289 log.cc:824] T 00000000000000000000000000000000 P 9103752b4bdc4023bc1ccf23d819db73: Log is configured to *not* fsync() on all Append() calls
I20260926 03:36:46.259466 21292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:46.245835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48177 (local address 127.0.0.1:35195)
0926 03:36:46.246696 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:46.246709 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:46.246757 (+ 48us) server_negotiation.cc:407] Connection header received
0926 03:36:46.249260 (+ 2503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:46.249287 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:46.249727 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:46.250077 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:46.250777 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:46.252239 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:46.254306 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:46.255082 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:46.255277 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:46.256712 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:46.256721 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:46.256737 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:36:46.257577 (+ 840us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:46.257706 (+ 129us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:36:46.257745 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:46.257875 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:46.258143 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 03:36:46.258426 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":416,"threads_started":1}
I20260926 03:36:46.259892 21289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9103752b4bdc4023bc1ccf23d819db73: No bootstrap required, opened a new log
I20260926 03:36:46.263427 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:36:46.270498 21289 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9103752b4bdc4023bc1ccf23d819db73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9103752b4bdc4023bc1ccf23d819db73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35195 } }
I20260926 03:36:46.271095 21289 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9103752b4bdc4023bc1ccf23d819db73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:36:46.271242 21289 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9103752b4bdc4023bc1ccf23d819db73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9103752b4bdc4023bc1ccf23d819db73, State: Initialized, Role: FOLLOWER
I20260926 03:36:46.271919 21289 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9103752b4bdc4023bc1ccf23d819db73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9103752b4bdc4023bc1ccf23d819db73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35195 } }
I20260926 03:36:46.272212 21289 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9103752b4bdc4023bc1ccf23d819db73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:36:46.272377 21289 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9103752b4bdc4023bc1ccf23d819db73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:36:46.272639 21289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9103752b4bdc4023bc1ccf23d819db73 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:36:46.276105 21289 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9103752b4bdc4023bc1ccf23d819db73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9103752b4bdc4023bc1ccf23d819db73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35195 } }
I20260926 03:36:46.276858 21289 leader_election.cc:302] T 00000000000000000000000000000000 P 9103752b4bdc4023bc1ccf23d819db73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9103752b4bdc4023bc1ccf23d819db73; no voters:
I20260926 03:36:46.277496 21292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:46.269559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43527 (local address 127.0.0.1:35195)
0926 03:36:46.269784 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:46.269790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:46.270062 (+ 272us) server_negotiation.cc:407] Connection header received
0926 03:36:46.270299 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:46.270304 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:46.270387 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:46.270481 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:46.271186 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:46.272247 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:46.274266 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:46.274761 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:46.274881 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:46.276242 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:46.276249 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:46.276254 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:36:46.276281 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:46.276329 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:36:46.276348 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:46.276433 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:46.276967 (+ 534us) server_negotiation.cc:299] Negotiation successful
0926 03:36:46.277117 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260926 03:36:46.277707 21289 leader_election.cc:288] T 00000000000000000000000000000000 P 9103752b4bdc4023bc1ccf23d819db73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:36:46.278373 21299 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9103752b4bdc4023bc1ccf23d819db73 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:36:46.279278 21299 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9103752b4bdc4023bc1ccf23d819db73 [term 1 LEADER]: Becoming Leader. State: Replica: 9103752b4bdc4023bc1ccf23d819db73, State: Running, Role: LEADER
I20260926 03:36:46.279899 21299 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9103752b4bdc4023bc1ccf23d819db73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9103752b4bdc4023bc1ccf23d819db73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35195 } }
I20260926 03:36:46.280253 21289 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9103752b4bdc4023bc1ccf23d819db73 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:36:46.283136 21300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9103752b4bdc4023bc1ccf23d819db73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9103752b4bdc4023bc1ccf23d819db73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9103752b4bdc4023bc1ccf23d819db73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35195 } } }
I20260926 03:36:46.283761 21300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9103752b4bdc4023bc1ccf23d819db73 [sys.catalog]: This master's current role is: LEADER
I20260926 03:36:46.284049 21301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9103752b4bdc4023bc1ccf23d819db73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9103752b4bdc4023bc1ccf23d819db73. Latest consensus state: current_term: 1 leader_uuid: "9103752b4bdc4023bc1ccf23d819db73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9103752b4bdc4023bc1ccf23d819db73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35195 } } }
I20260926 03:36:46.284372 21301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9103752b4bdc4023bc1ccf23d819db73 [sys.catalog]: This master's current role is: LEADER
W20260926 03:36:46.295060 21315 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9103752b4bdc4023bc1ccf23d819db73: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:36:46.295243 21315 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:36:46.295404 21306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:36:46.297319 21306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:36:46.307080 21306 catalog_manager.cc:1347] Generated new cluster ID: 13f376338f0c4649a7a0c49164bcc06b
I20260926 03:36:46.307274 21306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:36:46.322031 21306 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:36:46.323494 21306 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:36:46.336305 21306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9103752b4bdc4023bc1ccf23d819db73: Generated new TSK 0
I20260926 03:36:46.337414 21306 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:36:46.344684 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 21222
W20260926 03:36:46.345992 21284 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48177
W20260926 03:36:46.346801 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 21222: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:36:46.346961 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 21222
2026-09-26T03:36:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2039 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:36:46 dist-test-slave-dt7p krb5kdc[21320](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:36:46 dist-test-slave-dt7p krb5kdc[21320](info): set up 2 sockets
Sep 26 03:36:46 dist-test-slave-dt7p krb5kdc[21320](info): commencing operation
krb5kdc: starting...
W20260926 03:36:47.461328 388 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.001s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:36:47 dist-test-slave-dt7p krb5kdc[21320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:36:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:36:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:36:47.744717 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37345
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:36:48.128186 21333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:36:48.128548 21333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:36:48.128763 21333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:36:48.136252 21333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:36:48.136430 21333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:36:48.136538 21333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:36:48.136682 21333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:36:48.136767 21333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:36:48.146974 21333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:36:48.148520 21333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:36:48.150159 21333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:36:48.163357 21338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:48.163331 21341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:48.163261 21339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:36:48.164479 21333 server_base.cc:1034] running on GCE node
I20260926 03:36:48.165421 21333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:36:48.167034 21333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:36:48.168247 21333 hybrid_clock.cc:648] HybridClock initialized: now 1790393808168195 us; error 38 us; skew 500 ppm
I20260926 03:36:48.168699 21333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:36:48 dist-test-slave-dt7p krb5kdc[21320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:36:48.172094 21333 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:36:48.173697 21333 webserver.cc:460] Webserver started at http://127.0.0.1:41759/ using document root <none> and password file <none>
I20260926 03:36:48.174468 21333 fs_manager.cc:359] Metadata directory not provided
I20260926 03:36:48.174604 21333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:36:48.174953 21333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:36:48.178418 21333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "8a2a2772ff0943eeb5eb48fa6889b3c0"
format_stamp: "Formatted at 2026-09-26 03:36:48 on dist-test-slave-dt7p"
server_key: "1ac95e44067cd0ffc06363587a14d182"
server_key_iv: "cb536eb58e7f9ded67348ca77b4aaad5"
server_key_version: "encryptionkey@0"
I20260926 03:36:48.179234 21333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "8a2a2772ff0943eeb5eb48fa6889b3c0"
format_stamp: "Formatted at 2026-09-26 03:36:48 on dist-test-slave-dt7p"
server_key: "1ac95e44067cd0ffc06363587a14d182"
server_key_iv: "cb536eb58e7f9ded67348ca77b4aaad5"
server_key_version: "encryptionkey@0"
I20260926 03:36:48.185093 21333 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260926 03:36:48.188755 21348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:48.190141 21333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:36:48.190338 21333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "8a2a2772ff0943eeb5eb48fa6889b3c0"
format_stamp: "Formatted at 2026-09-26 03:36:48 on dist-test-slave-dt7p"
server_key: "1ac95e44067cd0ffc06363587a14d182"
server_key_iv: "cb536eb58e7f9ded67348ca77b4aaad5"
server_key_version: "encryptionkey@0"
I20260926 03:36:48.190532 21333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:36:48.196710 21333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:36:48.201287 21333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:36:48.201588 21333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:36:48.222581 21333 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37345
I20260926 03:36:48.222609 21399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37345 every 8 connection(s)
I20260926 03:36:48.224330 21333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:36:48.229110 21400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:36:48.232249 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 21333
I20260926 03:36:48.232502 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:36:48.233134 388 external_mini_cluster.cc:1428] Setting key 30e3746e2c56fad5ea494972503efba8
I20260926 03:36:48.243114 21403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:48.235649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59641 (local address 127.0.0.1:37345)
0926 03:36:48.236795 (+ 1146us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:48.236811 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:48.236870 (+ 59us) server_negotiation.cc:407] Connection header received
0926 03:36:48.238801 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:48.238830 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:48.239263 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:48.239736 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:48.241011 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:48.241018 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:48.241039 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:36:48.241055 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:48.241122 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:36:48.241153 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:48.241542 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:48.241711 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 03:36:48.241945 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":793,"thread_start_us":456,"threads_started":1}
I20260926 03:36:48.244882 21400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a2a2772ff0943eeb5eb48fa6889b3c0: Bootstrap starting.
I20260926 03:36:48.247706 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:36:48.248932 21400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a2a2772ff0943eeb5eb48fa6889b3c0: Neither blocks nor log segments found. Creating new log.
I20260926 03:36:48.250099 21400 log.cc:824] T 00000000000000000000000000000000 P 8a2a2772ff0943eeb5eb48fa6889b3c0: Log is configured to *not* fsync() on all Append() calls
I20260926 03:36:48.253993 21400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a2a2772ff0943eeb5eb48fa6889b3c0: No bootstrap required, opened a new log
I20260926 03:36:48.255616 21403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:48.252942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34647 (local address 127.0.0.1:37345)
0926 03:36:48.253147 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:48.253154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:48.253509 (+ 355us) server_negotiation.cc:407] Connection header received
0926 03:36:48.253680 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:48.253686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:48.253753 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:48.253870 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:48.254936 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:48.254942 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:48.254947 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:36:48.254973 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:48.255025 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:36:48.255047 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:48.255169 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:48.255249 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 03:36:48.255316 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 03:36:48.263151 21400 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a2a2772ff0943eeb5eb48fa6889b3c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a2a2772ff0943eeb5eb48fa6889b3c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37345 } }
I20260926 03:36:48.263592 21400 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a2a2772ff0943eeb5eb48fa6889b3c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:36:48.263710 21400 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a2a2772ff0943eeb5eb48fa6889b3c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2a2772ff0943eeb5eb48fa6889b3c0, State: Initialized, Role: FOLLOWER
I20260926 03:36:48.264348 21400 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a2a2772ff0943eeb5eb48fa6889b3c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a2a2772ff0943eeb5eb48fa6889b3c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37345 } }
I20260926 03:36:48.264622 21400 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a2a2772ff0943eeb5eb48fa6889b3c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:36:48.264786 21400 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a2a2772ff0943eeb5eb48fa6889b3c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:36:48.264985 21400 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a2a2772ff0943eeb5eb48fa6889b3c0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:36:48.267413 21400 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a2a2772ff0943eeb5eb48fa6889b3c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a2a2772ff0943eeb5eb48fa6889b3c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37345 } }
I20260926 03:36:48.268059 21400 leader_election.cc:302] T 00000000000000000000000000000000 P 8a2a2772ff0943eeb5eb48fa6889b3c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a2a2772ff0943eeb5eb48fa6889b3c0; no voters:
I20260926 03:36:48.268764 21400 leader_election.cc:288] T 00000000000000000000000000000000 P 8a2a2772ff0943eeb5eb48fa6889b3c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:36:48.268913 21410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a2a2772ff0943eeb5eb48fa6889b3c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:36:48.269726 21410 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a2a2772ff0943eeb5eb48fa6889b3c0 [term 1 LEADER]: Becoming Leader. State: Replica: 8a2a2772ff0943eeb5eb48fa6889b3c0, State: Running, Role: LEADER
I20260926 03:36:48.270340 21410 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a2a2772ff0943eeb5eb48fa6889b3c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a2a2772ff0943eeb5eb48fa6889b3c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37345 } }
I20260926 03:36:48.271080 21400 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a2a2772ff0943eeb5eb48fa6889b3c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:36:48.273866 21411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a2a2772ff0943eeb5eb48fa6889b3c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a2a2772ff0943eeb5eb48fa6889b3c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a2a2772ff0943eeb5eb48fa6889b3c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37345 } } }
I20260926 03:36:48.274339 21411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a2a2772ff0943eeb5eb48fa6889b3c0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:36:48.274377 21412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a2a2772ff0943eeb5eb48fa6889b3c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a2a2772ff0943eeb5eb48fa6889b3c0. Latest consensus state: current_term: 1 leader_uuid: "8a2a2772ff0943eeb5eb48fa6889b3c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a2a2772ff0943eeb5eb48fa6889b3c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37345 } } }
I20260926 03:36:48.274649 21412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a2a2772ff0943eeb5eb48fa6889b3c0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:36:48.276607 21419 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:36:48.281700 21419 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:36:48.291769 21419 catalog_manager.cc:1347] Generated new cluster ID: 59b12be359564384b6b41aec43e32508
I20260926 03:36:48.291932 21419 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:36:48.304034 21419 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:36:48.304672 21419 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:36:48.312402 21419 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a2a2772ff0943eeb5eb48fa6889b3c0: Generated new TSK 0
I20260926 03:36:48.313674 21419 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:36:48.326022 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 21333
W20260926 03:36:48.327292 21395 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59641
W20260926 03:36:48.328047 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 21333: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:36:48.328218 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 21333
2026-09-26T03:36:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1980 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:36:48 dist-test-slave-dt7p krb5kdc[21431](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:36:48 dist-test-slave-dt7p krb5kdc[21431](info): set up 2 sockets
Sep 26 03:36:48 dist-test-slave-dt7p krb5kdc[21431](info): commencing operation
krb5kdc: starting...
W20260926 03:36:49.441524 388 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.001s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:36:49 dist-test-slave-dt7p krb5kdc[21431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:36:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:36:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:36:49.724645 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39025
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:36:50.124181 21444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:36:50.124631 21444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:36:50.124845 21444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:36:50.132598 21444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:36:50.132779 21444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:36:50.132895 21444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:36:50.133014 21444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:36:50.133105 21444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:36:50.143668 21444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:36:50.145339 21444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:36:50.146893 21444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:36:50.160358 21449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:50.160336 21450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:50.160526 21452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:36:50.162065 21444 server_base.cc:1034] running on GCE node
I20260926 03:36:50.163381 21444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:36:50.165917 21444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:36:50.167204 21444 hybrid_clock.cc:648] HybridClock initialized: now 1790393810167129 us; error 54 us; skew 500 ppm
I20260926 03:36:50.167766 21444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:36:50 dist-test-slave-dt7p krb5kdc[21431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:36:50.177145 21444 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:36:50.178784 21444 webserver.cc:460] Webserver started at http://127.0.0.1:45047/ using document root <none> and password file <none>
I20260926 03:36:50.179605 21444 fs_manager.cc:359] Metadata directory not provided
I20260926 03:36:50.179711 21444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:36:50.180006 21444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:36:50.183606 21444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "5d3120ec03fd4bd9bbffe1568b2ad9b4"
format_stamp: "Formatted at 2026-09-26 03:36:50 on dist-test-slave-dt7p"
server_key: "710bad36a37d99d559278aabcd37a6d4"
server_key_iv: "01450158c73d31ce9d26d36538548095"
server_key_version: "encryptionkey@0"
I20260926 03:36:50.184410 21444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "5d3120ec03fd4bd9bbffe1568b2ad9b4"
format_stamp: "Formatted at 2026-09-26 03:36:50 on dist-test-slave-dt7p"
server_key: "710bad36a37d99d559278aabcd37a6d4"
server_key_iv: "01450158c73d31ce9d26d36538548095"
server_key_version: "encryptionkey@0"
I20260926 03:36:50.190436 21444 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260926 03:36:50.194211 21459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:50.195755 21444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:36:50.195981 21444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "5d3120ec03fd4bd9bbffe1568b2ad9b4"
format_stamp: "Formatted at 2026-09-26 03:36:50 on dist-test-slave-dt7p"
server_key: "710bad36a37d99d559278aabcd37a6d4"
server_key_iv: "01450158c73d31ce9d26d36538548095"
server_key_version: "encryptionkey@0"
I20260926 03:36:50.196251 21444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:36:50.235206 21444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:36:50.240621 21444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:36:50.240986 21444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:36:50.260849 21444 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39025
I20260926 03:36:50.260869 21510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39025 every 8 connection(s)
I20260926 03:36:50.262650 21444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:36:50.264294 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 21444
I20260926 03:36:50.264539 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:36:50.265226 388 external_mini_cluster.cc:1428] Setting key 5b21871c8957b3ff730da081e71d8cfe
I20260926 03:36:50.268397 21511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 03:36:50 dist-test-slave-dt7p krb5kdc[21431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 03:36:50.283664 21511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d3120ec03fd4bd9bbffe1568b2ad9b4: Bootstrap starting.
I20260926 03:36:50.288551 21511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d3120ec03fd4bd9bbffe1568b2ad9b4: Neither blocks nor log segments found. Creating new log.
I20260926 03:36:50.290133 21511 log.cc:824] T 00000000000000000000000000000000 P 5d3120ec03fd4bd9bbffe1568b2ad9b4: Log is configured to *not* fsync() on all Append() calls
I20260926 03:36:50.292956 21514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:50.267497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:36679 (local address 127.0.0.1:39025)
0926 03:36:50.268399 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:50.268418 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:50.268481 (+ 63us) server_negotiation.cc:407] Connection header received
0926 03:36:50.271249 (+ 2768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:50.271283 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:50.271858 (+ 575us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:50.272400 (+ 542us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:50.274017 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:50.276206 (+ 2189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:50.277698 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:50.278775 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:50.278998 (+ 223us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:50.283901 (+ 4903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:50.283913 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:50.283931 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:36:50.283985 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:50.288059 (+ 4074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:50.289117 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:50.289128 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:50.289143 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:50.289379 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:50.289842 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:50.289851 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:50.289857 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:50.290577 (+ 720us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:50.291011 (+ 434us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:50.291421 (+ 410us) server_negotiation.cc:299] Negotiation successful
0926 03:36:50.291782 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":360,"threads_started":1}
I20260926 03:36:50.293882 21511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d3120ec03fd4bd9bbffe1568b2ad9b4: No bootstrap required, opened a new log
I20260926 03:36:50.297302 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:36:50.302978 21511 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d3120ec03fd4bd9bbffe1568b2ad9b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d3120ec03fd4bd9bbffe1568b2ad9b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39025 } }
I20260926 03:36:50.303598 21511 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d3120ec03fd4bd9bbffe1568b2ad9b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:36:50.303732 21511 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d3120ec03fd4bd9bbffe1568b2ad9b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d3120ec03fd4bd9bbffe1568b2ad9b4, State: Initialized, Role: FOLLOWER
I20260926 03:36:50.304595 21511 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d3120ec03fd4bd9bbffe1568b2ad9b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d3120ec03fd4bd9bbffe1568b2ad9b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39025 } }
I20260926 03:36:50.304956 21511 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d3120ec03fd4bd9bbffe1568b2ad9b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:36:50.305115 21511 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d3120ec03fd4bd9bbffe1568b2ad9b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:36:50.305356 21511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d3120ec03fd4bd9bbffe1568b2ad9b4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:36:50.308769 21511 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d3120ec03fd4bd9bbffe1568b2ad9b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d3120ec03fd4bd9bbffe1568b2ad9b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39025 } }
I20260926 03:36:50.309520 21511 leader_election.cc:302] T 00000000000000000000000000000000 P 5d3120ec03fd4bd9bbffe1568b2ad9b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d3120ec03fd4bd9bbffe1568b2ad9b4; no voters:
I20260926 03:36:50.310468 21511 leader_election.cc:288] T 00000000000000000000000000000000 P 5d3120ec03fd4bd9bbffe1568b2ad9b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:36:50.310544 21521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d3120ec03fd4bd9bbffe1568b2ad9b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:36:50.312239 21521 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d3120ec03fd4bd9bbffe1568b2ad9b4 [term 1 LEADER]: Becoming Leader. State: Replica: 5d3120ec03fd4bd9bbffe1568b2ad9b4, State: Running, Role: LEADER
I20260926 03:36:50.312950 21521 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d3120ec03fd4bd9bbffe1568b2ad9b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d3120ec03fd4bd9bbffe1568b2ad9b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39025 } }
I20260926 03:36:50.313460 21511 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d3120ec03fd4bd9bbffe1568b2ad9b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:36:50.316215 21522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d3120ec03fd4bd9bbffe1568b2ad9b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d3120ec03fd4bd9bbffe1568b2ad9b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d3120ec03fd4bd9bbffe1568b2ad9b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39025 } } }
I20260926 03:36:50.316690 21522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d3120ec03fd4bd9bbffe1568b2ad9b4 [sys.catalog]: This master's current role is: LEADER
I20260926 03:36:50.317183 21523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d3120ec03fd4bd9bbffe1568b2ad9b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d3120ec03fd4bd9bbffe1568b2ad9b4. Latest consensus state: current_term: 1 leader_uuid: "5d3120ec03fd4bd9bbffe1568b2ad9b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d3120ec03fd4bd9bbffe1568b2ad9b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39025 } } }
I20260926 03:36:50.317507 21523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d3120ec03fd4bd9bbffe1568b2ad9b4 [sys.catalog]: This master's current role is: LEADER
I20260926 03:36:50.321650 21526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:36:50.323009 21514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:50.302427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:48917 (local address 127.0.0.1:39025)
0926 03:36:50.302682 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:50.302690 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:50.303055 (+ 365us) server_negotiation.cc:407] Connection header received
0926 03:36:50.303248 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:50.303252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:50.303328 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:50.303421 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:50.305580 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:50.306489 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:50.308378 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:50.308992 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:50.309100 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:50.312919 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:50.312925 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:50.312930 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:36:50.312975 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:50.317988 (+ 5013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:50.319351 (+ 1363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:50.319361 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:50.319368 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:50.319675 (+ 307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:50.320420 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:50.320428 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:50.320435 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:50.321052 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:50.321435 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:50.322308 (+ 873us) server_negotiation.cc:299] Negotiation successful
0926 03:36:50.322524 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 03:36:50.340632 21526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:36:50.350112 21526 catalog_manager.cc:1347] Generated new cluster ID: 1d3455c11c6b44f2a7019b0c395f983f
I20260926 03:36:50.350241 21526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:36:50.374452 21526 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:36:50.375967 21526 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:36:50.386377 21526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d3120ec03fd4bd9bbffe1568b2ad9b4: Generated new TSK 0
I20260926 03:36:50.387400 21526 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:36:50.397131 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 21444
I20260926 03:36:50.398427 21506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:36:50.504681 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 21444
W20260926 03:36:50.524614 21429 connection.cc:535] client connection to 127.0.0.1:39025 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39025): Connection reset by peer (error 104)
2026-09-26T03:36:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2178 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:36:50 dist-test-slave-dt7p krb5kdc[21543](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:36:50 dist-test-slave-dt7p krb5kdc[21543](info): set up 2 sockets
Sep 26 03:36:50 dist-test-slave-dt7p krb5kdc[21543](info): commencing operation
krb5kdc: starting...
W20260926 03:36:51.625375 388 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:36:51 dist-test-slave-dt7p krb5kdc[21543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:36:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:36:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:36:51.920459 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44161
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:36:52.309545 21556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:36:52.309908 21556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:36:52.310012 21556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:36:52.317714 21556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:36:52.317868 21556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:36:52.317925 21556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:36:52.317994 21556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:36:52.318041 21556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:36:52.328789 21556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:36:52.330463 21556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:36:52.331982 21556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:36:52.344930 21562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:52.344898 21564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:52.345274 21561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:36:52.346397 21556 server_base.cc:1034] running on GCE node
I20260926 03:36:52.347342 21556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:36:52.349126 21556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:36:52.350380 21556 hybrid_clock.cc:648] HybridClock initialized: now 1790393812350335 us; error 43 us; skew 500 ppm
I20260926 03:36:52.350775 21556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:36:52 dist-test-slave-dt7p krb5kdc[21543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:36:52.354431 21556 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:36:52.356113 21556 webserver.cc:460] Webserver started at http://127.0.0.1:37247/ using document root <none> and password file <none>
I20260926 03:36:52.356964 21556 fs_manager.cc:359] Metadata directory not provided
I20260926 03:36:52.357075 21556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:36:52.357379 21556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:36:52.360880 21556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "55515cb808b5465292542ae5a3cccfb5"
format_stamp: "Formatted at 2026-09-26 03:36:52 on dist-test-slave-dt7p"
server_key: "a6f3743063d6203539ca5da38bf2c977"
server_key_iv: "98490e59c5ff2536bab4ba8a002fb30c"
server_key_version: "encryptionkey@0"
I20260926 03:36:52.361681 21556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "55515cb808b5465292542ae5a3cccfb5"
format_stamp: "Formatted at 2026-09-26 03:36:52 on dist-test-slave-dt7p"
server_key: "a6f3743063d6203539ca5da38bf2c977"
server_key_iv: "98490e59c5ff2536bab4ba8a002fb30c"
server_key_version: "encryptionkey@0"
I20260926 03:36:52.367551 21556 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 03:36:52.371832 21571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:52.373639 21556 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260926 03:36:52.373814 21556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "55515cb808b5465292542ae5a3cccfb5"
format_stamp: "Formatted at 2026-09-26 03:36:52 on dist-test-slave-dt7p"
server_key: "a6f3743063d6203539ca5da38bf2c977"
server_key_iv: "98490e59c5ff2536bab4ba8a002fb30c"
server_key_version: "encryptionkey@0"
I20260926 03:36:52.374018 21556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:36:52.394390 21556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:36:52.399341 21556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:36:52.399621 21556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:36:52.420517 21556 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44161
I20260926 03:36:52.420543 21622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44161 every 8 connection(s)
I20260926 03:36:52.422304 21556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:36:52.426781 21623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:36:52.431720 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 21556
I20260926 03:36:52.431954 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:36:52.432469 388 external_mini_cluster.cc:1428] Setting key 8cd95e1a49fc0a1f13e07789a1d8e35d
I20260926 03:36:52.439283 21623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55515cb808b5465292542ae5a3cccfb5: Bootstrap starting.
I20260926 03:36:52.443678 21623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55515cb808b5465292542ae5a3cccfb5: Neither blocks nor log segments found. Creating new log.
I20260926 03:36:52.445344 21623 log.cc:824] T 00000000000000000000000000000000 P 55515cb808b5465292542ae5a3cccfb5: Log is configured to *not* fsync() on all Append() calls
I20260926 03:36:52.448894 21626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:52.434633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:38017 (local address 127.0.0.1:44161)
0926 03:36:52.435502 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:52.435515 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:52.435563 (+ 48us) server_negotiation.cc:407] Connection header received
0926 03:36:52.437869 (+ 2306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:52.437896 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:52.438353 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:52.438720 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:52.439387 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:52.440916 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:52.443080 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:52.444018 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:52.444187 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:52.445795 (+ 1608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:52.445804 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:52.445820 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:36:52.446792 (+ 972us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:52.446911 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:36:52.446950 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:52.447123 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:52.447469 (+ 346us) server_negotiation.cc:299] Negotiation successful
0926 03:36:52.447692 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":374,"threads_started":1}
I20260926 03:36:52.449301 21623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55515cb808b5465292542ae5a3cccfb5: No bootstrap required, opened a new log
I20260926 03:36:52.453403 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:36:52.458389 21623 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55515cb808b5465292542ae5a3cccfb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55515cb808b5465292542ae5a3cccfb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44161 } }
I20260926 03:36:52.459007 21623 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55515cb808b5465292542ae5a3cccfb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:36:52.459156 21623 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55515cb808b5465292542ae5a3cccfb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55515cb808b5465292542ae5a3cccfb5, State: Initialized, Role: FOLLOWER
I20260926 03:36:52.459865 21623 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55515cb808b5465292542ae5a3cccfb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55515cb808b5465292542ae5a3cccfb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44161 } }
I20260926 03:36:52.460141 21623 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55515cb808b5465292542ae5a3cccfb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:36:52.460315 21623 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55515cb808b5465292542ae5a3cccfb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:36:52.460520 21623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55515cb808b5465292542ae5a3cccfb5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:36:52.463070 21623 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55515cb808b5465292542ae5a3cccfb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55515cb808b5465292542ae5a3cccfb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44161 } }
I20260926 03:36:52.463696 21623 leader_election.cc:302] T 00000000000000000000000000000000 P 55515cb808b5465292542ae5a3cccfb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55515cb808b5465292542ae5a3cccfb5; no voters:
I20260926 03:36:52.464517 21623 leader_election.cc:288] T 00000000000000000000000000000000 P 55515cb808b5465292542ae5a3cccfb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:36:52.464656 21633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55515cb808b5465292542ae5a3cccfb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:36:52.465497 21626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:52.458359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:59845 (local address 127.0.0.1:44161)
0926 03:36:52.458573 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:52.458582 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:52.458957 (+ 375us) server_negotiation.cc:407] Connection header received
0926 03:36:52.459195 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:52.459199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:52.459291 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:52.459386 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:52.459930 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:52.461004 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:52.462466 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:52.462961 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:52.463105 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:52.464384 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:52.464389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:52.464392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:36:52.464416 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:52.464486 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:36:52.464506 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:52.464683 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:52.464975 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 03:36:52.465044 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 03:36:52.466276 21633 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55515cb808b5465292542ae5a3cccfb5 [term 1 LEADER]: Becoming Leader. State: Replica: 55515cb808b5465292542ae5a3cccfb5, State: Running, Role: LEADER
I20260926 03:36:52.466956 21633 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55515cb808b5465292542ae5a3cccfb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55515cb808b5465292542ae5a3cccfb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44161 } }
I20260926 03:36:52.467528 21623 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55515cb808b5465292542ae5a3cccfb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:36:52.470325 21635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55515cb808b5465292542ae5a3cccfb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55515cb808b5465292542ae5a3cccfb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55515cb808b5465292542ae5a3cccfb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44161 } } }
I20260926 03:36:52.470815 21635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55515cb808b5465292542ae5a3cccfb5 [sys.catalog]: This master's current role is: LEADER
I20260926 03:36:52.470431 21634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55515cb808b5465292542ae5a3cccfb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55515cb808b5465292542ae5a3cccfb5. Latest consensus state: current_term: 1 leader_uuid: "55515cb808b5465292542ae5a3cccfb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55515cb808b5465292542ae5a3cccfb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44161 } } }
I20260926 03:36:52.471264 21634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55515cb808b5465292542ae5a3cccfb5 [sys.catalog]: This master's current role is: LEADER
I20260926 03:36:52.480002 21640 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:36:52.485184 21640 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:36:52.496845 21640 catalog_manager.cc:1347] Generated new cluster ID: 34f70dc8265e4a8299d035830e989d9e
I20260926 03:36:52.496990 21640 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:36:52.513335 21640 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:36:52.515383 21640 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:36:52.529345 21640 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55515cb808b5465292542ae5a3cccfb5: Generated new TSK 0
I20260926 03:36:52.530712 21640 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:36:52.542455 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 21556
W20260926 03:36:52.544245 21618 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.164:38017
W20260926 03:36:52.545202 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 21556: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:36:52.545356 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 21556
W20260926 03:36:52.565445 21541 connection.cc:535] client connection to 127.0.0.1:44161 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44161): Connection reset by peer (error 104)
2026-09-26T03:36:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2040 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:36:52 dist-test-slave-dt7p krb5kdc[21654](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:36:52 dist-test-slave-dt7p krb5kdc[21654](info): set up 2 sockets
Sep 26 03:36:52 dist-test-slave-dt7p krb5kdc[21654](info): commencing operation
krb5kdc: starting...
W20260926 03:36:53.665424 388 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:36:53 dist-test-slave-dt7p krb5kdc[21654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:36:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:36:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:36:53.952253 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35939
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:36:54.347836 21667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:36:54.348151 21667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:36:54.348248 21667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:36:54.355440 21667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:36:54.355543 21667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:36:54.355590 21667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:36:54.355661 21667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:36:54.355706 21667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:36:54.365363 21667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:36:54.366817 21667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:36:54.368407 21667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:36:54.381570 21672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:54.381697 21673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:54.381600 21675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:36:54.382606 21667 server_base.cc:1034] running on GCE node
I20260926 03:36:54.383463 21667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:36:54.385176 21667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:36:54.386422 21667 hybrid_clock.cc:648] HybridClock initialized: now 1790393814386386 us; error 57 us; skew 500 ppm
I20260926 03:36:54.386826 21667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:36:54 dist-test-slave-dt7p krb5kdc[21654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:36:54.390457 21667 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:36:54.392144 21667 webserver.cc:460] Webserver started at http://127.0.0.1:34669/ using document root <none> and password file <none>
I20260926 03:36:54.393013 21667 fs_manager.cc:359] Metadata directory not provided
I20260926 03:36:54.393175 21667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:36:54.393522 21667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:36:54.397466 21667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "d7cbe6ebacc642cbb0ead969f8cada47"
format_stamp: "Formatted at 2026-09-26 03:36:54 on dist-test-slave-dt7p"
server_key: "b4306e84e381f968774ab857499f4856"
server_key_iv: "12713aa8a8415311cb0b885d270884d9"
server_key_version: "encryptionkey@0"
I20260926 03:36:54.398422 21667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "d7cbe6ebacc642cbb0ead969f8cada47"
format_stamp: "Formatted at 2026-09-26 03:36:54 on dist-test-slave-dt7p"
server_key: "b4306e84e381f968774ab857499f4856"
server_key_iv: "12713aa8a8415311cb0b885d270884d9"
server_key_version: "encryptionkey@0"
I20260926 03:36:54.404147 21667 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:36:54.408053 21682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:54.409595 21667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 03:36:54.409785 21667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "d7cbe6ebacc642cbb0ead969f8cada47"
format_stamp: "Formatted at 2026-09-26 03:36:54 on dist-test-slave-dt7p"
server_key: "b4306e84e381f968774ab857499f4856"
server_key_iv: "12713aa8a8415311cb0b885d270884d9"
server_key_version: "encryptionkey@0"
I20260926 03:36:54.409989 21667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:36:54.416101 21667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:36:54.420650 21667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:36:54.420964 21667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:36:54.441850 21667 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35939
I20260926 03:36:54.441900 21733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35939 every 8 connection(s)
I20260926 03:36:54.443445 21667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:36:54.447631 21734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:36:54.448863 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 21667
I20260926 03:36:54.449069 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:36:54.449576 388 external_mini_cluster.cc:1428] Setting key 9e1a44aec9abd3425d60927d63b5627c
I20260926 03:36:54.457892 21737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:54.451520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:38465 (local address 127.0.0.1:35939)
0926 03:36:54.452385 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:54.452399 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:54.452448 (+ 49us) server_negotiation.cc:407] Connection header received
0926 03:36:54.454287 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:54.454311 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:54.454692 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:54.455061 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:54.456207 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:54.456214 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:54.456227 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:36:54.456241 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:54.456309 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:36:54.456329 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:54.456500 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:54.456697 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 03:36:54.456940 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":401,"threads_started":1}
I20260926 03:36:54.461441 21734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7cbe6ebacc642cbb0ead969f8cada47: Bootstrap starting.
I20260926 03:36:54.461623 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:36:54.464831 21734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7cbe6ebacc642cbb0ead969f8cada47: Neither blocks nor log segments found. Creating new log.
I20260926 03:36:54.466203 21734 log.cc:824] T 00000000000000000000000000000000 P d7cbe6ebacc642cbb0ead969f8cada47: Log is configured to *not* fsync() on all Append() calls
I20260926 03:36:54.469072 21737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:54.466479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:41505 (local address 127.0.0.1:35939)
0926 03:36:54.466647 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:54.466670 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:54.467111 (+ 441us) server_negotiation.cc:407] Connection header received
0926 03:36:54.467302 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:54.467308 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:54.467370 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:54.467482 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:54.468420 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:54.468425 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:54.468430 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:36:54.468448 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:54.468496 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:36:54.468512 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:54.468662 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:54.468749 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 03:36:54.468812 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 03:36:54.469861 21734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7cbe6ebacc642cbb0ead969f8cada47: No bootstrap required, opened a new log
I20260926 03:36:54.477509 21734 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7cbe6ebacc642cbb0ead969f8cada47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7cbe6ebacc642cbb0ead969f8cada47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35939 } }
I20260926 03:36:54.477888 21734 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7cbe6ebacc642cbb0ead969f8cada47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:36:54.478021 21734 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7cbe6ebacc642cbb0ead969f8cada47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7cbe6ebacc642cbb0ead969f8cada47, State: Initialized, Role: FOLLOWER
I20260926 03:36:54.478655 21734 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7cbe6ebacc642cbb0ead969f8cada47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7cbe6ebacc642cbb0ead969f8cada47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35939 } }
I20260926 03:36:54.478930 21734 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7cbe6ebacc642cbb0ead969f8cada47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:36:54.479086 21734 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7cbe6ebacc642cbb0ead969f8cada47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:36:54.479297 21734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7cbe6ebacc642cbb0ead969f8cada47 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:36:54.481513 21734 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7cbe6ebacc642cbb0ead969f8cada47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7cbe6ebacc642cbb0ead969f8cada47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35939 } }
I20260926 03:36:54.482008 21734 leader_election.cc:302] T 00000000000000000000000000000000 P d7cbe6ebacc642cbb0ead969f8cada47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7cbe6ebacc642cbb0ead969f8cada47; no voters:
I20260926 03:36:54.482614 21734 leader_election.cc:288] T 00000000000000000000000000000000 P d7cbe6ebacc642cbb0ead969f8cada47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:36:54.482795 21744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7cbe6ebacc642cbb0ead969f8cada47 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:36:54.483500 21744 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7cbe6ebacc642cbb0ead969f8cada47 [term 1 LEADER]: Becoming Leader. State: Replica: d7cbe6ebacc642cbb0ead969f8cada47, State: Running, Role: LEADER
I20260926 03:36:54.484066 21744 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7cbe6ebacc642cbb0ead969f8cada47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7cbe6ebacc642cbb0ead969f8cada47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35939 } }
I20260926 03:36:54.484836 21734 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7cbe6ebacc642cbb0ead969f8cada47 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:36:54.488842 21745 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7cbe6ebacc642cbb0ead969f8cada47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7cbe6ebacc642cbb0ead969f8cada47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7cbe6ebacc642cbb0ead969f8cada47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35939 } } }
I20260926 03:36:54.489753 21745 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7cbe6ebacc642cbb0ead969f8cada47 [sys.catalog]: This master's current role is: LEADER
I20260926 03:36:54.489332 21746 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7cbe6ebacc642cbb0ead969f8cada47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7cbe6ebacc642cbb0ead969f8cada47. Latest consensus state: current_term: 1 leader_uuid: "d7cbe6ebacc642cbb0ead969f8cada47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7cbe6ebacc642cbb0ead969f8cada47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35939 } } }
I20260926 03:36:54.490406 21746 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7cbe6ebacc642cbb0ead969f8cada47 [sys.catalog]: This master's current role is: LEADER
I20260926 03:36:54.491411 21757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:36:54.496862 21757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:36:54.505353 21757 catalog_manager.cc:1347] Generated new cluster ID: e0db3b7646ea4ee8846d64541b63f92d
I20260926 03:36:54.505457 21757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:36:54.518893 21757 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:36:54.519402 21757 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:36:54.531826 21757 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7cbe6ebacc642cbb0ead969f8cada47: Generated new TSK 0
I20260926 03:36:54.532868 21757 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:36:54.540217 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 21667
W20260926 03:36:54.541740 21729 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.164:38465
W20260926 03:36:54.542466 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 21667: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:36:54.542611 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 21667
2026-09-26T03:36:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1997 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:36:54 dist-test-slave-dt7p krb5kdc[21765](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:36:54 dist-test-slave-dt7p krb5kdc[21765](info): set up 2 sockets
Sep 26 03:36:54 dist-test-slave-dt7p krb5kdc[21765](info): commencing operation
krb5kdc: starting...
W20260926 03:36:55.673506 388 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.003s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:36:55 dist-test-slave-dt7p krb5kdc[21765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:36:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:36:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:36:55.970319 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34245
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:36:56.356789 21778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:36:56.357131 21778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:36:56.357331 21778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:36:56.364876 21778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:36:56.364984 21778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:36:56.365106 21778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:36:56.365228 21778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:36:56.365310 21778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:36:56.375537 21778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34245
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:36:56.377200 21778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:36:56.378803 21778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:36:56.391896 21784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:56.391899 21783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:56.391840 21786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:36:56.393350 21778 server_base.cc:1034] running on GCE node
I20260926 03:36:56.394253 21778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:36:56.395839 21778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:36:56.397073 21778 hybrid_clock.cc:648] HybridClock initialized: now 1790393816396996 us; error 73 us; skew 500 ppm
I20260926 03:36:56.397472 21778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:36:56 dist-test-slave-dt7p krb5kdc[21765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:36:56.413183 21778 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:36:56.415021 21778 webserver.cc:460] Webserver started at http://127.0.0.1:38969/ using document root <none> and password file <none>
I20260926 03:36:56.415817 21778 fs_manager.cc:359] Metadata directory not provided
I20260926 03:36:56.415934 21778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:36:56.416240 21778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:36:56.419775 21778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "6ad28f9426fe43c19d8d263f7c56f021"
format_stamp: "Formatted at 2026-09-26 03:36:56 on dist-test-slave-dt7p"
server_key: "4b1464d1e7ec8158b5ae63ced1566225"
server_key_iv: "bf565541b11554c5a27dc4109ba9a34e"
server_key_version: "encryptionkey@0"
I20260926 03:36:56.420557 21778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "6ad28f9426fe43c19d8d263f7c56f021"
format_stamp: "Formatted at 2026-09-26 03:36:56 on dist-test-slave-dt7p"
server_key: "4b1464d1e7ec8158b5ae63ced1566225"
server_key_iv: "bf565541b11554c5a27dc4109ba9a34e"
server_key_version: "encryptionkey@0"
I20260926 03:36:56.426364 21778 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:36:56.430423 21793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:56.431955 21778 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:36:56.432157 21778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "6ad28f9426fe43c19d8d263f7c56f021"
format_stamp: "Formatted at 2026-09-26 03:36:56 on dist-test-slave-dt7p"
server_key: "4b1464d1e7ec8158b5ae63ced1566225"
server_key_iv: "bf565541b11554c5a27dc4109ba9a34e"
server_key_version: "encryptionkey@0"
I20260926 03:36:56.432387 21778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:36:56.456751 21778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:36:56.461679 21778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:36:56.462008 21778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:36:56.484123 21778 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34245
I20260926 03:36:56.484144 21844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34245 every 8 connection(s)
I20260926 03:36:56.485991 21778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:36:56.490513 21845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:36:56.491050 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 21778
I20260926 03:36:56.491269 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:36:56.491900 388 external_mini_cluster.cc:1428] Setting key 613e4efbcdc6ab729f8449e4fb7c480f
I20260926 03:36:56.505856 21845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ad28f9426fe43c19d8d263f7c56f021: Bootstrap starting.
Sep 26 03:36:56 dist-test-slave-dt7p krb5kdc[21765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 03:36:56.509914 21845 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ad28f9426fe43c19d8d263f7c56f021: Neither blocks nor log segments found. Creating new log.
I20260926 03:36:56.511281 21845 log.cc:824] T 00000000000000000000000000000000 P 6ad28f9426fe43c19d8d263f7c56f021: Log is configured to *not* fsync() on all Append() calls
I20260926 03:36:56.514607 21845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ad28f9426fe43c19d8d263f7c56f021: No bootstrap required, opened a new log
I20260926 03:36:56.518625 21848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:56.494037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:42345 (local address 127.0.0.1:34245)
0926 03:36:56.494907 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:56.494920 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:56.494969 (+ 49us) server_negotiation.cc:407] Connection header received
0926 03:36:56.497393 (+ 2424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:56.497422 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:56.497887 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:56.498255 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:56.500202 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:56.501630 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:56.503513 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:56.504189 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:56.504356 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:56.509545 (+ 5189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:56.509553 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:56.509568 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:36:56.509615 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:56.513380 (+ 3765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:56.514218 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:56.514225 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:56.514232 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:56.514397 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:56.515053 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:56.515057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:56.515060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:56.515913 (+ 853us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:56.516293 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:56.516651 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 03:36:56.517051 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":398,"threads_started":1}
I20260926 03:36:56.522926 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:36:56.524808 21845 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ad28f9426fe43c19d8d263f7c56f021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad28f9426fe43c19d8d263f7c56f021" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34245 } }
I20260926 03:36:56.525309 21845 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ad28f9426fe43c19d8d263f7c56f021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:36:56.525416 21845 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ad28f9426fe43c19d8d263f7c56f021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ad28f9426fe43c19d8d263f7c56f021, State: Initialized, Role: FOLLOWER
I20260926 03:36:56.526127 21845 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ad28f9426fe43c19d8d263f7c56f021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad28f9426fe43c19d8d263f7c56f021" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34245 } }
I20260926 03:36:56.526487 21845 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ad28f9426fe43c19d8d263f7c56f021 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:36:56.526669 21845 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ad28f9426fe43c19d8d263f7c56f021 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:36:56.526861 21845 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ad28f9426fe43c19d8d263f7c56f021 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:36:56.530278 21845 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ad28f9426fe43c19d8d263f7c56f021 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad28f9426fe43c19d8d263f7c56f021" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34245 } }
I20260926 03:36:56.530890 21845 leader_election.cc:302] T 00000000000000000000000000000000 P 6ad28f9426fe43c19d8d263f7c56f021 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ad28f9426fe43c19d8d263f7c56f021; no voters:
I20260926 03:36:56.531550 21845 leader_election.cc:288] T 00000000000000000000000000000000 P 6ad28f9426fe43c19d8d263f7c56f021 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:36:56.531761 21855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ad28f9426fe43c19d8d263f7c56f021 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:36:56.534097 21845 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ad28f9426fe43c19d8d263f7c56f021 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:36:56.538570 21855 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ad28f9426fe43c19d8d263f7c56f021 [term 1 LEADER]: Becoming Leader. State: Replica: 6ad28f9426fe43c19d8d263f7c56f021, State: Running, Role: LEADER
I20260926 03:36:56.539167 21855 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ad28f9426fe43c19d8d263f7c56f021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad28f9426fe43c19d8d263f7c56f021" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34245 } }
I20260926 03:36:56.542589 21857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ad28f9426fe43c19d8d263f7c56f021 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ad28f9426fe43c19d8d263f7c56f021. Latest consensus state: current_term: 1 leader_uuid: "6ad28f9426fe43c19d8d263f7c56f021" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad28f9426fe43c19d8d263f7c56f021" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34245 } } }
I20260926 03:36:56.543020 21857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ad28f9426fe43c19d8d263f7c56f021 [sys.catalog]: This master's current role is: LEADER
I20260926 03:36:56.543311 21856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ad28f9426fe43c19d8d263f7c56f021 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ad28f9426fe43c19d8d263f7c56f021" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad28f9426fe43c19d8d263f7c56f021" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34245 } } }
I20260926 03:36:56.543679 21856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ad28f9426fe43c19d8d263f7c56f021 [sys.catalog]: This master's current role is: LEADER
I20260926 03:36:56.552346 21848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:56.528960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:43137 (local address 127.0.0.1:34245)
0926 03:36:56.529236 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:56.529243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:56.529649 (+ 406us) server_negotiation.cc:407] Connection header received
0926 03:36:56.529802 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:56.529808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:56.529895 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:56.530015 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:56.531862 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:56.533215 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:56.535819 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:56.536933 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:56.537110 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:56.541606 (+ 4496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:56.541616 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:56.541625 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:36:56.541682 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:56.546901 (+ 5219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:56.548141 (+ 1240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:56.548150 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:56.548157 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:56.548368 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:36:56.549082 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:36:56.549089 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:36:56.549092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:36:56.549675 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:56.550009 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:56.551509 (+ 1500us) server_negotiation.cc:299] Negotiation successful
0926 03:36:56.551780 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260926 03:36:56.557900 21867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:36:56.564771 21867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:36:56.574488 21867 catalog_manager.cc:1347] Generated new cluster ID: 24854ff562324ed8a9d825eb64c994e8
I20260926 03:36:56.574608 21867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:36:56.599159 21867 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:36:56.600545 21867 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:36:56.607569 21867 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ad28f9426fe43c19d8d263f7c56f021: Generated new TSK 0
I20260926 03:36:56.608466 21867 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:36:56.616747 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 21778
I20260926 03:36:56.617892 21840 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:36:56.716780 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 21778
W20260926 03:36:56.736812 21763 connection.cc:535] client connection to 127.0.0.1:34245 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34245): Connection reset by peer (error 104)
2026-09-26T03:36:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2173 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:36:56 dist-test-slave-dt7p krb5kdc[21877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:36:56 dist-test-slave-dt7p krb5kdc[21877](info): set up 2 sockets
Sep 26 03:36:56 dist-test-slave-dt7p krb5kdc[21877](info): commencing operation
krb5kdc: starting...
W20260926 03:36:57.829461 388 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.004s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:36:58 dist-test-slave-dt7p krb5kdc[21877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:36:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:36:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:36:58.136821 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44177
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:36:58.517043 21890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:36:58.517385 21890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:36:58.517554 21890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:36:58.525146 21890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:36:58.525274 21890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:36:58.525357 21890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:36:58.525473 21890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:36:58.525555 21890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:36:58.536253 21890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44177
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:36:58.537958 21890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:36:58.539592 21890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:36:58.552860 21898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:58.552956 21895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:36:58.552959 21896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:36:58.558166 21890 server_base.cc:1034] running on GCE node
I20260926 03:36:58.559023 21890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:36:58.560736 21890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:36:58.561969 21890 hybrid_clock.cc:648] HybridClock initialized: now 1790393818561929 us; error 34 us; skew 500 ppm
I20260926 03:36:58.562386 21890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:36:58 dist-test-slave-dt7p krb5kdc[21877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:36:58.565914 21890 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:36:58.567602 21890 webserver.cc:460] Webserver started at http://127.0.0.1:42969/ using document root <none> and password file <none>
I20260926 03:36:58.568396 21890 fs_manager.cc:359] Metadata directory not provided
I20260926 03:36:58.568543 21890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:36:58.568910 21890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:36:58.572171 21890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "e3600ae9599f432e86f255365ba65bb5"
format_stamp: "Formatted at 2026-09-26 03:36:58 on dist-test-slave-dt7p"
server_key: "c729dc9283b5406337ec6d32509dddfb"
server_key_iv: "f9283073d878a0332e6756f453444a5a"
server_key_version: "encryptionkey@0"
I20260926 03:36:58.573073 21890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "e3600ae9599f432e86f255365ba65bb5"
format_stamp: "Formatted at 2026-09-26 03:36:58 on dist-test-slave-dt7p"
server_key: "c729dc9283b5406337ec6d32509dddfb"
server_key_iv: "f9283073d878a0332e6756f453444a5a"
server_key_version: "encryptionkey@0"
I20260926 03:36:58.578536 21890 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:36:58.582170 21905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:36:58.583603 21890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:36:58.583791 21890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "e3600ae9599f432e86f255365ba65bb5"
format_stamp: "Formatted at 2026-09-26 03:36:58 on dist-test-slave-dt7p"
server_key: "c729dc9283b5406337ec6d32509dddfb"
server_key_iv: "f9283073d878a0332e6756f453444a5a"
server_key_version: "encryptionkey@0"
I20260926 03:36:58.583993 21890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:36:58.613456 21890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:36:58.617794 21890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:36:58.618119 21890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:36:58.638651 21890 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44177
I20260926 03:36:58.638684 21956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44177 every 8 connection(s)
I20260926 03:36:58.640221 21890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:36:58.644764 21957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:36:58.646126 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 21890
I20260926 03:36:58.646361 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:36:58.646849 388 external_mini_cluster.cc:1428] Setting key ed03f6b8a99f6a491dc647187ab7f7d1
I20260926 03:36:58.661213 21957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3600ae9599f432e86f255365ba65bb5: Bootstrap starting.
I20260926 03:36:58.663465 21960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:58.649040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:45289 (local address 127.0.0.1:44177)
0926 03:36:58.650007 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:58.650025 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:58.650086 (+ 61us) server_negotiation.cc:407] Connection header received
0926 03:36:58.652658 (+ 2572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:58.652690 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:58.653276 (+ 586us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:58.653759 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:58.654312 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:58.656282 (+ 1970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:58.657733 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:58.658874 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:58.659066 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:58.660308 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:58.660318 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:58.660337 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:36:58.661125 (+ 788us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:58.661245 (+ 120us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:36:58.661281 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:58.661440 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:58.661709 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 03:36:58.661910 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":388,"threads_started":1}
I20260926 03:36:58.665597 21957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3600ae9599f432e86f255365ba65bb5: Neither blocks nor log segments found. Creating new log.
I20260926 03:36:58.666889 21957 log.cc:824] T 00000000000000000000000000000000 P e3600ae9599f432e86f255365ba65bb5: Log is configured to *not* fsync() on all Append() calls
I20260926 03:36:58.667536 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:36:58.669946 21957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3600ae9599f432e86f255365ba65bb5: No bootstrap required, opened a new log
I20260926 03:36:58.679730 21960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:36:58.672606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:45815 (local address 127.0.0.1:44177)
0926 03:36:58.672861 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 03:36:58.672867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:36:58.673249 (+ 382us) server_negotiation.cc:407] Connection header received
0926 03:36:58.673420 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:36:58.673423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:36:58.673493 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:36:58.673577 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:36:58.674266 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:58.675067 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:58.677110 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:36:58.677572 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:36:58.677672 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:36:58.678935 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:36:58.678939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:36:58.678943 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:36:58.678959 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:36:58.679011 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:36:58.679027 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:36:58.679111 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 03:36:58.679321 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 03:36:58.679367 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 03:36:58.680296 21957 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3600ae9599f432e86f255365ba65bb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3600ae9599f432e86f255365ba65bb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44177 } }
I20260926 03:36:58.680825 21957 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3600ae9599f432e86f255365ba65bb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:36:58.680951 21957 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3600ae9599f432e86f255365ba65bb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3600ae9599f432e86f255365ba65bb5, State: Initialized, Role: FOLLOWER
I20260926 03:36:58.681656 21957 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3600ae9599f432e86f255365ba65bb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3600ae9599f432e86f255365ba65bb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44177 } }
I20260926 03:36:58.681905 21957 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3600ae9599f432e86f255365ba65bb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:36:58.682080 21957 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3600ae9599f432e86f255365ba65bb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:36:58.682257 21957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3600ae9599f432e86f255365ba65bb5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:36:58.684803 21957 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3600ae9599f432e86f255365ba65bb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3600ae9599f432e86f255365ba65bb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44177 } }
I20260926 03:36:58.685391 21957 leader_election.cc:302] T 00000000000000000000000000000000 P e3600ae9599f432e86f255365ba65bb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3600ae9599f432e86f255365ba65bb5; no voters:
I20260926 03:36:58.686012 21957 leader_election.cc:288] T 00000000000000000000000000000000 P e3600ae9599f432e86f255365ba65bb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:36:58.686152 21967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3600ae9599f432e86f255365ba65bb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:36:58.686955 21967 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3600ae9599f432e86f255365ba65bb5 [term 1 LEADER]: Becoming Leader. State: Replica: e3600ae9599f432e86f255365ba65bb5, State: Running, Role: LEADER
I20260926 03:36:58.687674 21967 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3600ae9599f432e86f255365ba65bb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3600ae9599f432e86f255365ba65bb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44177 } }
I20260926 03:36:58.688721 21957 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3600ae9599f432e86f255365ba65bb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:36:58.690774 21968 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3600ae9599f432e86f255365ba65bb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3600ae9599f432e86f255365ba65bb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3600ae9599f432e86f255365ba65bb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44177 } } }
I20260926 03:36:58.691180 21968 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3600ae9599f432e86f255365ba65bb5 [sys.catalog]: This master's current role is: LEADER
I20260926 03:36:58.691287 21969 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3600ae9599f432e86f255365ba65bb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3600ae9599f432e86f255365ba65bb5. Latest consensus state: current_term: 1 leader_uuid: "e3600ae9599f432e86f255365ba65bb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3600ae9599f432e86f255365ba65bb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44177 } } }
I20260926 03:36:58.691596 21969 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3600ae9599f432e86f255365ba65bb5 [sys.catalog]: This master's current role is: LEADER
I20260926 03:36:58.698288 21972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:36:58.704237 21972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:36:58.714411 21972 catalog_manager.cc:1347] Generated new cluster ID: 1659d1d4f11443629720557666ec6137
I20260926 03:36:58.714553 21972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:36:58.725979 21972 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:36:58.727396 21972 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:36:58.734118 21972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3600ae9599f432e86f255365ba65bb5: Generated new TSK 0
I20260926 03:36:58.735090 21972 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:36:58.747466 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 21890
W20260926 03:36:58.748775 21952 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.164:45289
W20260926 03:36:58.749706 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 21890: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:36:58.749843 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 21890
W20260926 03:36:58.770923 21875 connection.cc:535] client connection to 127.0.0.1:44177 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44177): Connection reset by peer (error 104)
2026-09-26T03:36:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2034 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:36:58 dist-test-slave-dt7p krb5kdc[21988](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:36:58 dist-test-slave-dt7p krb5kdc[21988](info): set up 2 sockets
Sep 26 03:36:58 dist-test-slave-dt7p krb5kdc[21988](info): commencing operation
krb5kdc: starting...
W20260926 03:36:59.869365 388 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.003s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:37:00 dist-test-slave-dt7p krb5kdc[21988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:37:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:37:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:37:00.179591 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42561
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:37:00.566738 22001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:37:00.567129 22001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:37:00.567303 22001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:37:00.574853 22001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:37:00.575021 22001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:37:00.575147 22001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:37:00.575263 22001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:37:00.575344 22001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:37:00.585714 22001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42561
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393463219689-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:37:00.587261 22001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:37:00.588824 22001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:37:00.601774 22006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:00.601728 22007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:00.601728 22009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:00.603096 22001 server_base.cc:1034] running on GCE node
I20260926 03:37:00.603974 22001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:37:00.605578 22001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:37:00.606813 22001 hybrid_clock.cc:648] HybridClock initialized: now 1790393820606767 us; error 41 us; skew 500 ppm
I20260926 03:37:00.607178 22001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:37:00 dist-test-slave-dt7p krb5kdc[21988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:37:00.610718 22001 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:37:00.612353 22001 webserver.cc:460] Webserver started at http://127.0.0.1:45111/ using document root <none> and password file <none>
I20260926 03:37:00.613175 22001 fs_manager.cc:359] Metadata directory not provided
I20260926 03:37:00.613323 22001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:37:00.613659 22001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:37:00.616976 22001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "000ce54450fb48bc9759c5d0737049ae"
format_stamp: "Formatted at 2026-09-26 03:37:00 on dist-test-slave-dt7p"
server_key: "d2faed202014146df4399aba95e67400"
server_key_iv: "d1451deb3bc621ce58b83308b226ea75"
server_key_version: "encryptionkey@0"
I20260926 03:37:00.617790 22001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "000ce54450fb48bc9759c5d0737049ae"
format_stamp: "Formatted at 2026-09-26 03:37:00 on dist-test-slave-dt7p"
server_key: "d2faed202014146df4399aba95e67400"
server_key_iv: "d1451deb3bc621ce58b83308b226ea75"
server_key_version: "encryptionkey@0"
I20260926 03:37:00.623152 22001 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:37:00.626925 22016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:00.628382 22001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:37:00.628648 22001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "000ce54450fb48bc9759c5d0737049ae"
format_stamp: "Formatted at 2026-09-26 03:37:00 on dist-test-slave-dt7p"
server_key: "d2faed202014146df4399aba95e67400"
server_key_iv: "d1451deb3bc621ce58b83308b226ea75"
server_key_version: "encryptionkey@0"
I20260926 03:37:00.628862 22001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:37:00.635131 22001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:37:00.639935 22001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:37:00.640290 22001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:37:00.661819 22001 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42561
I20260926 03:37:00.661854 22067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42561 every 8 connection(s)
I20260926 03:37:00.663532 22001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:37:00.668004 22068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:00.669735 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 22001
I20260926 03:37:00.669953 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:37:00.670596 388 external_mini_cluster.cc:1428] Setting key f8d0c70a0a3e3e47de13b090bfcc5e2a
I20260926 03:37:00.681979 22071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:00.673102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:42217 (local address 127.0.0.1:42561)
0926 03:37:00.674108 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:00.674124 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:00.674176 (+ 52us) server_negotiation.cc:407] Connection header received
0926 03:37:00.676983 (+ 2807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:00.677021 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:00.677594 (+ 573us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:00.678134 (+ 540us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:00.679563 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:00.679574 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:00.679593 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:37:00.679619 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:00.679729 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:37:00.679773 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:00.680042 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:00.680182 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 03:37:00.680425 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":755,"thread_start_us":487,"threads_started":1}
I20260926 03:37:00.686199 22068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 000ce54450fb48bc9759c5d0737049ae: Bootstrap starting.
I20260926 03:37:00.686710 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:37:00.690814 22068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 000ce54450fb48bc9759c5d0737049ae: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:00.692377 22068 log.cc:824] T 00000000000000000000000000000000 P 000ce54450fb48bc9759c5d0737049ae: Log is configured to *not* fsync() on all Append() calls
I20260926 03:37:00.697168 22071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:00.694996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:53891 (local address 127.0.0.1:42561)
0926 03:37:00.695215 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:00.695221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:00.695246 (+ 25us) server_negotiation.cc:407] Connection header received
0926 03:37:00.695322 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:00.695328 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:00.695396 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:00.695527 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:00.696492 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:00.696497 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:00.696500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:37:00.696514 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:00.696554 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:37:00.696635 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:00.696756 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:00.696894 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 03:37:00.696945 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:37:00.697775 22068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 000ce54450fb48bc9759c5d0737049ae: No bootstrap required, opened a new log
I20260926 03:37:00.706203 22068 raft_consensus.cc:348] T 00000000000000000000000000000000 P 000ce54450fb48bc9759c5d0737049ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "000ce54450fb48bc9759c5d0737049ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42561 } }
I20260926 03:37:00.706624 22068 raft_consensus.cc:374] T 00000000000000000000000000000000 P 000ce54450fb48bc9759c5d0737049ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:00.706743 22068 raft_consensus.cc:729] T 00000000000000000000000000000000 P 000ce54450fb48bc9759c5d0737049ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 000ce54450fb48bc9759c5d0737049ae, State: Initialized, Role: FOLLOWER
I20260926 03:37:00.707340 22068 consensus_queue.cc:260] T 00000000000000000000000000000000 P 000ce54450fb48bc9759c5d0737049ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "000ce54450fb48bc9759c5d0737049ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42561 } }
I20260926 03:37:00.707604 22068 raft_consensus.cc:388] T 00000000000000000000000000000000 P 000ce54450fb48bc9759c5d0737049ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:37:00.707765 22068 raft_consensus.cc:482] T 00000000000000000000000000000000 P 000ce54450fb48bc9759c5d0737049ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:37:00.707940 22068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 000ce54450fb48bc9759c5d0737049ae [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:00.710361 22068 raft_consensus.cc:504] T 00000000000000000000000000000000 P 000ce54450fb48bc9759c5d0737049ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "000ce54450fb48bc9759c5d0737049ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42561 } }
I20260926 03:37:00.710992 22068 leader_election.cc:302] T 00000000000000000000000000000000 P 000ce54450fb48bc9759c5d0737049ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 000ce54450fb48bc9759c5d0737049ae; no voters:
I20260926 03:37:00.711700 22068 leader_election.cc:288] T 00000000000000000000000000000000 P 000ce54450fb48bc9759c5d0737049ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:37:00.711829 22078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 000ce54450fb48bc9759c5d0737049ae [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:37:00.712960 22078 raft_consensus.cc:686] T 00000000000000000000000000000000 P 000ce54450fb48bc9759c5d0737049ae [term 1 LEADER]: Becoming Leader. State: Replica: 000ce54450fb48bc9759c5d0737049ae, State: Running, Role: LEADER
I20260926 03:37:00.713557 22078 consensus_queue.cc:237] T 00000000000000000000000000000000 P 000ce54450fb48bc9759c5d0737049ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "000ce54450fb48bc9759c5d0737049ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42561 } }
I20260926 03:37:00.714100 22068 sys_catalog.cc:564] T 00000000000000000000000000000000 P 000ce54450fb48bc9759c5d0737049ae [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:37:00.717198 22079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 000ce54450fb48bc9759c5d0737049ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "000ce54450fb48bc9759c5d0737049ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "000ce54450fb48bc9759c5d0737049ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42561 } } }
I20260926 03:37:00.717686 22079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 000ce54450fb48bc9759c5d0737049ae [sys.catalog]: This master's current role is: LEADER
I20260926 03:37:00.717051 22080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 000ce54450fb48bc9759c5d0737049ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 000ce54450fb48bc9759c5d0737049ae. Latest consensus state: current_term: 1 leader_uuid: "000ce54450fb48bc9759c5d0737049ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "000ce54450fb48bc9759c5d0737049ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42561 } } }
I20260926 03:37:00.718228 22080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 000ce54450fb48bc9759c5d0737049ae [sys.catalog]: This master's current role is: LEADER
I20260926 03:37:00.720443 22087 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:37:00.725068 22087 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:37:00.734155 22087 catalog_manager.cc:1347] Generated new cluster ID: afeca2ae06ae4617ad3e298bad15e7ff
I20260926 03:37:00.734262 22087 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:37:00.749104 22087 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:37:00.749642 22087 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:37:00.758827 22087 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 000ce54450fb48bc9759c5d0737049ae: Generated new TSK 0
I20260926 03:37:00.760125 22087 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:37:00.771293 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 22001
W20260926 03:37:00.772717 22063 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.164:42217
W20260926 03:37:00.773614 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 22001: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:37:00.773785 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 22001
2026-09-26T03:37:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2022 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:37:00 dist-test-slave-dt7p krb5kdc[22099](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:37:00 dist-test-slave-dt7p krb5kdc[22099](info): set up 2 sockets
Sep 26 03:37:00 dist-test-slave-dt7p krb5kdc[22099](info): commencing operation
krb5kdc: starting...
W20260926 03:37:01.893404 388 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.003s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:37:02 dist-test-slave-dt7p krb5kdc[22099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:37:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:37:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:37:02.185402 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36685
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:37:02.574088 22112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:37:02.574430 22112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:37:02.574532 22112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:37:02.582343 22112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:37:02.582487 22112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:37:02.582540 22112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:37:02.582612 22112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:37:02.582660 22112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:37:02.593428 22112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36685
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:37:02.594976 22112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:37:02.596509 22112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:37:02.609266 22120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:02.609295 22117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:02.609340 22118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:02.610507 22112 server_base.cc:1034] running on GCE node
I20260926 03:37:02.611348 22112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:37:02.613081 22112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:37:02.614292 22112 hybrid_clock.cc:648] HybridClock initialized: now 1790393822614254 us; error 30 us; skew 500 ppm
I20260926 03:37:02.614674 22112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:37:02 dist-test-slave-dt7p krb5kdc[22099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:37:02.623272 22112 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:37:02.624960 22112 webserver.cc:460] Webserver started at http://127.0.97.62:34491/ using document root <none> and password file <none>
I20260926 03:37:02.625718 22112 fs_manager.cc:359] Metadata directory not provided
I20260926 03:37:02.625828 22112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:37:02.626121 22112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:37:02.629930 22112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "451facaab6f844f082b9ea97a0b3b2d2"
format_stamp: "Formatted at 2026-09-26 03:37:02 on dist-test-slave-dt7p"
server_key: "634cc3488dc1654d18d7d800816210b0"
server_key_iv: "ca71380f3a57bcef0641d9978a183724"
server_key_version: "encryptionkey@0"
I20260926 03:37:02.630716 22112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "451facaab6f844f082b9ea97a0b3b2d2"
format_stamp: "Formatted at 2026-09-26 03:37:02 on dist-test-slave-dt7p"
server_key: "634cc3488dc1654d18d7d800816210b0"
server_key_iv: "ca71380f3a57bcef0641d9978a183724"
server_key_version: "encryptionkey@0"
I20260926 03:37:02.636348 22112 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 03:37:02.640120 22127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:02.641717 22112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:37:02.641894 22112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "451facaab6f844f082b9ea97a0b3b2d2"
format_stamp: "Formatted at 2026-09-26 03:37:02 on dist-test-slave-dt7p"
server_key: "634cc3488dc1654d18d7d800816210b0"
server_key_iv: "ca71380f3a57bcef0641d9978a183724"
server_key_version: "encryptionkey@0"
I20260926 03:37:02.642096 22112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:37:02.688417 22112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:37:02.693542 22112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:37:02.693871 22112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:37:02.714813 22112 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36685
I20260926 03:37:02.714846 22178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36685 every 8 connection(s)
I20260926 03:37:02.716382 22112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:37:02.720934 22179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:02.726352 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 22112
I20260926 03:37:02.726572 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:37:02.727234 388 external_mini_cluster.cc:1428] Setting key 4966e962a7eb4f6732fdf22aab483a9a
I20260926 03:37:02.733523 22179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 451facaab6f844f082b9ea97a0b3b2d2: Bootstrap starting.
I20260926 03:37:02.737655 22179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 451facaab6f844f082b9ea97a0b3b2d2: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:02.738978 22179 log.cc:824] T 00000000000000000000000000000000 P 451facaab6f844f082b9ea97a0b3b2d2: Log is configured to *not* fsync() on all Append() calls
I20260926 03:37:02.742794 22179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 451facaab6f844f082b9ea97a0b3b2d2: No bootstrap required, opened a new log
Sep 26 03:37:02 dist-test-slave-dt7p krb5kdc[22099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:37:02.753417 22179 raft_consensus.cc:348] T 00000000000000000000000000000000 P 451facaab6f844f082b9ea97a0b3b2d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451facaab6f844f082b9ea97a0b3b2d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36685 } }
I20260926 03:37:02.753796 22182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:02.729257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55521 (local address 127.0.97.62:36685)
0926 03:37:02.730220 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:02.730233 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:02.730288 (+ 55us) server_negotiation.cc:407] Connection header received
0926 03:37:02.732124 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:02.732149 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:02.732629 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:02.732997 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:02.734905 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:02.736317 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:02.738174 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:02.738887 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:02.739025 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:02.744385 (+ 5360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:02.744394 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:02.744413 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:02.744460 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:02.748709 (+ 4249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:02.749551 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:02.749560 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:02.749570 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:02.749815 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:02.750246 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:02.750250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:02.750254 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:02.751069 (+ 815us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:02.751439 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:02.751881 (+ 442us) server_negotiation.cc:299] Negotiation successful
0926 03:37:02.752265 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":441,"threads_started":1}
I20260926 03:37:02.753958 22179 raft_consensus.cc:374] T 00000000000000000000000000000000 P 451facaab6f844f082b9ea97a0b3b2d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:02.754106 22179 raft_consensus.cc:729] T 00000000000000000000000000000000 P 451facaab6f844f082b9ea97a0b3b2d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451facaab6f844f082b9ea97a0b3b2d2, State: Initialized, Role: FOLLOWER
I20260926 03:37:02.754767 22179 consensus_queue.cc:260] T 00000000000000000000000000000000 P 451facaab6f844f082b9ea97a0b3b2d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451facaab6f844f082b9ea97a0b3b2d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36685 } }
I20260926 03:37:02.755048 22179 raft_consensus.cc:388] T 00000000000000000000000000000000 P 451facaab6f844f082b9ea97a0b3b2d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:37:02.755223 22179 raft_consensus.cc:482] T 00000000000000000000000000000000 P 451facaab6f844f082b9ea97a0b3b2d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:37:02.755438 22179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 451facaab6f844f082b9ea97a0b3b2d2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:02.758507 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:37:02.758757 22179 raft_consensus.cc:504] T 00000000000000000000000000000000 P 451facaab6f844f082b9ea97a0b3b2d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451facaab6f844f082b9ea97a0b3b2d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36685 } }
I20260926 03:37:02.759354 22179 leader_election.cc:302] T 00000000000000000000000000000000 P 451facaab6f844f082b9ea97a0b3b2d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 451facaab6f844f082b9ea97a0b3b2d2; no voters:
I20260926 03:37:02.760063 22179 leader_election.cc:288] T 00000000000000000000000000000000 P 451facaab6f844f082b9ea97a0b3b2d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:37:02.760188 22184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 451facaab6f844f082b9ea97a0b3b2d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:37:02.762504 22179 sys_catalog.cc:564] T 00000000000000000000000000000000 P 451facaab6f844f082b9ea97a0b3b2d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:37:02.765285 22184 raft_consensus.cc:686] T 00000000000000000000000000000000 P 451facaab6f844f082b9ea97a0b3b2d2 [term 1 LEADER]: Becoming Leader. State: Replica: 451facaab6f844f082b9ea97a0b3b2d2, State: Running, Role: LEADER
I20260926 03:37:02.765930 22184 consensus_queue.cc:237] T 00000000000000000000000000000000 P 451facaab6f844f082b9ea97a0b3b2d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451facaab6f844f082b9ea97a0b3b2d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36685 } }
I20260926 03:37:02.768937 22185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 451facaab6f844f082b9ea97a0b3b2d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "451facaab6f844f082b9ea97a0b3b2d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451facaab6f844f082b9ea97a0b3b2d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36685 } } }
I20260926 03:37:02.769344 22185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 451facaab6f844f082b9ea97a0b3b2d2 [sys.catalog]: This master's current role is: LEADER
I20260926 03:37:02.769872 22186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 451facaab6f844f082b9ea97a0b3b2d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 451facaab6f844f082b9ea97a0b3b2d2. Latest consensus state: current_term: 1 leader_uuid: "451facaab6f844f082b9ea97a0b3b2d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451facaab6f844f082b9ea97a0b3b2d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36685 } } }
I20260926 03:37:02.770243 22186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 451facaab6f844f082b9ea97a0b3b2d2 [sys.catalog]: This master's current role is: LEADER
I20260926 03:37:02.772848 22192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:37:02.780821 22192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:37:02.804226 22192 catalog_manager.cc:1347] Generated new cluster ID: bd554cec784a40bc950e32703979dc54
I20260926 03:37:02.804358 22192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:37:02.808455 22182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:02.781105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43713 (local address 127.0.97.62:36685)
0926 03:37:02.781370 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:02.781384 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:02.786341 (+ 4957us) server_negotiation.cc:407] Connection header received
0926 03:37:02.786613 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:02.786622 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:02.786756 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:02.786905 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:02.790030 (+ 3125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:02.791294 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:02.793331 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:02.794078 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:02.794183 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:02.798076 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:02.798084 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:02.798090 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:02.798140 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:02.802866 (+ 4726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:02.803983 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:02.803989 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:02.803996 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:02.804153 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:02.804906 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:02.804914 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:02.804920 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:02.805535 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:02.807360 (+ 1825us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:02.807600 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 03:37:02.807869 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 03:37:02.815327 22192 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:37:02.817598 22192 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:37:02.834542 22192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 451facaab6f844f082b9ea97a0b3b2d2: Generated new TSK 0
I20260926 03:37:02.835508 22192 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:37:02.843814 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 22112
I20260926 03:37:02.845275 22174 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:37:02.947480 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 22112
W20260926 03:37:02.967547 22097 connection.cc:535] client connection to 127.0.97.62:36685 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:36685): Connection reset by peer (error 104)
2026-09-26T03:37:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2173 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:37:03 dist-test-slave-dt7p krb5kdc[22212](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:37:03 dist-test-slave-dt7p krb5kdc[22212](info): set up 2 sockets
Sep 26 03:37:03 dist-test-slave-dt7p krb5kdc[22212](info): commencing operation
krb5kdc: starting...
W20260926 03:37:04.073406 388 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.001s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:37:04 dist-test-slave-dt7p krb5kdc[22212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:37:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:37:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:37:04.398232 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42921
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:37:04.789927 22225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:37:04.790301 22225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:37:04.790491 22225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:37:04.798354 22225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:37:04.798480 22225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:37:04.798585 22225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:37:04.798699 22225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:37:04.798794 22225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:37:04.810487 22225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42921
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:37:04.812188 22225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:37:04.813849 22225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:37:04.827571 22231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:04.827690 22230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:04.827666 22233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:04.828532 22225 server_base.cc:1034] running on GCE node
I20260926 03:37:04.829424 22225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:37:04.831028 22225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:37:04.832263 22225 hybrid_clock.cc:648] HybridClock initialized: now 1790393824832219 us; error 42 us; skew 500 ppm
I20260926 03:37:04.832722 22225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:37:04 dist-test-slave-dt7p krb5kdc[22212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:37:04.836143 22225 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:37:04.837834 22225 webserver.cc:460] Webserver started at http://127.0.97.62:43213/ using document root <none> and password file <none>
I20260926 03:37:04.838600 22225 fs_manager.cc:359] Metadata directory not provided
I20260926 03:37:04.838718 22225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:37:04.839020 22225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:37:04.842564 22225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "daffe8af83c140cc8a3fc401563316e3"
format_stamp: "Formatted at 2026-09-26 03:37:04 on dist-test-slave-dt7p"
server_key: "85704420aafc39597bd5c574a29494fa"
server_key_iv: "9ca84b2100be3990032063368ae0e078"
server_key_version: "encryptionkey@0"
I20260926 03:37:04.843420 22225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "daffe8af83c140cc8a3fc401563316e3"
format_stamp: "Formatted at 2026-09-26 03:37:04 on dist-test-slave-dt7p"
server_key: "85704420aafc39597bd5c574a29494fa"
server_key_iv: "9ca84b2100be3990032063368ae0e078"
server_key_version: "encryptionkey@0"
I20260926 03:37:04.849649 22225 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260926 03:37:04.853776 22240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:04.855408 22225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260926 03:37:04.855635 22225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "daffe8af83c140cc8a3fc401563316e3"
format_stamp: "Formatted at 2026-09-26 03:37:04 on dist-test-slave-dt7p"
server_key: "85704420aafc39597bd5c574a29494fa"
server_key_iv: "9ca84b2100be3990032063368ae0e078"
server_key_version: "encryptionkey@0"
I20260926 03:37:04.855911 22225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:37:04.876639 22225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:37:04.881337 22225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:37:04.881605 22225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:37:04.903216 22225 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42921
I20260926 03:37:04.903236 22291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42921 every 8 connection(s)
I20260926 03:37:04.904930 22225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:37:04.909375 22292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:04.910539 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 22225
I20260926 03:37:04.910748 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:37:04.911394 388 external_mini_cluster.cc:1428] Setting key af5a6e0a80d6137351ffef5e88bebed0
I20260926 03:37:04.924845 22292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daffe8af83c140cc8a3fc401563316e3: Bootstrap starting.
I20260926 03:37:04.926342 22295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:04.913495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57033 (local address 127.0.97.62:42921)
0926 03:37:04.914386 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:04.914400 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:04.914450 (+ 50us) server_negotiation.cc:407] Connection header received
0926 03:37:04.916362 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:04.916390 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:04.916924 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:04.917303 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:04.917911 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:04.919276 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:04.921353 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:04.922121 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:04.922302 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:04.923751 (+ 1449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:04.923760 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:04.923779 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:37:04.924415 (+ 636us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:04.924501 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:37:04.924528 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:04.924757 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:04.925088 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 03:37:04.925277 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":430,"threads_started":1}
I20260926 03:37:04.928966 22292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P daffe8af83c140cc8a3fc401563316e3: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:04.930496 22292 log.cc:824] T 00000000000000000000000000000000 P daffe8af83c140cc8a3fc401563316e3: Log is configured to *not* fsync() on all Append() calls
I20260926 03:37:04.931047 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:37:04.934001 22292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daffe8af83c140cc8a3fc401563316e3: No bootstrap required, opened a new log
I20260926 03:37:04.943939 22295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:04.936632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53847 (local address 127.0.97.62:42921)
0926 03:37:04.936872 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:04.936882 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:04.937263 (+ 381us) server_negotiation.cc:407] Connection header received
0926 03:37:04.937510 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:04.937515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:04.937593 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:04.937722 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:04.938274 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:04.939198 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:04.940717 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:04.941462 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:04.941617 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:04.943065 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:04.943072 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:04.943077 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:37:04.943102 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:04.943152 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:37:04.943172 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:04.943299 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:04.943515 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 03:37:04.943582 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 03:37:04.943881 22292 raft_consensus.cc:348] T 00000000000000000000000000000000 P daffe8af83c140cc8a3fc401563316e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daffe8af83c140cc8a3fc401563316e3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42921 } }
I20260926 03:37:04.944545 22292 raft_consensus.cc:374] T 00000000000000000000000000000000 P daffe8af83c140cc8a3fc401563316e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:04.944738 22292 raft_consensus.cc:729] T 00000000000000000000000000000000 P daffe8af83c140cc8a3fc401563316e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daffe8af83c140cc8a3fc401563316e3, State: Initialized, Role: FOLLOWER
I20260926 03:37:04.945583 22292 consensus_queue.cc:260] T 00000000000000000000000000000000 P daffe8af83c140cc8a3fc401563316e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daffe8af83c140cc8a3fc401563316e3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42921 } }
I20260926 03:37:04.945899 22292 raft_consensus.cc:388] T 00000000000000000000000000000000 P daffe8af83c140cc8a3fc401563316e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:37:04.946058 22292 raft_consensus.cc:482] T 00000000000000000000000000000000 P daffe8af83c140cc8a3fc401563316e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:37:04.946247 22292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P daffe8af83c140cc8a3fc401563316e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:04.948778 22292 raft_consensus.cc:504] T 00000000000000000000000000000000 P daffe8af83c140cc8a3fc401563316e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daffe8af83c140cc8a3fc401563316e3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42921 } }
I20260926 03:37:04.949388 22292 leader_election.cc:302] T 00000000000000000000000000000000 P daffe8af83c140cc8a3fc401563316e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: daffe8af83c140cc8a3fc401563316e3; no voters:
I20260926 03:37:04.950049 22292 leader_election.cc:288] T 00000000000000000000000000000000 P daffe8af83c140cc8a3fc401563316e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:37:04.950261 22302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P daffe8af83c140cc8a3fc401563316e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:37:04.951057 22302 raft_consensus.cc:686] T 00000000000000000000000000000000 P daffe8af83c140cc8a3fc401563316e3 [term 1 LEADER]: Becoming Leader. State: Replica: daffe8af83c140cc8a3fc401563316e3, State: Running, Role: LEADER
I20260926 03:37:04.951822 22302 consensus_queue.cc:237] T 00000000000000000000000000000000 P daffe8af83c140cc8a3fc401563316e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daffe8af83c140cc8a3fc401563316e3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42921 } }
I20260926 03:37:04.952734 22292 sys_catalog.cc:564] T 00000000000000000000000000000000 P daffe8af83c140cc8a3fc401563316e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:37:04.955426 22304 sys_catalog.cc:455] T 00000000000000000000000000000000 P daffe8af83c140cc8a3fc401563316e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader daffe8af83c140cc8a3fc401563316e3. Latest consensus state: current_term: 1 leader_uuid: "daffe8af83c140cc8a3fc401563316e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daffe8af83c140cc8a3fc401563316e3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42921 } } }
I20260926 03:37:04.955989 22304 sys_catalog.cc:458] T 00000000000000000000000000000000 P daffe8af83c140cc8a3fc401563316e3 [sys.catalog]: This master's current role is: LEADER
I20260926 03:37:04.955459 22303 sys_catalog.cc:455] T 00000000000000000000000000000000 P daffe8af83c140cc8a3fc401563316e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "daffe8af83c140cc8a3fc401563316e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daffe8af83c140cc8a3fc401563316e3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42921 } } }
I20260926 03:37:04.956679 22303 sys_catalog.cc:458] T 00000000000000000000000000000000 P daffe8af83c140cc8a3fc401563316e3 [sys.catalog]: This master's current role is: LEADER
I20260926 03:37:04.961161 22308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:37:04.968604 22308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:37:04.980856 22308 catalog_manager.cc:1347] Generated new cluster ID: 911d9205b8d14f17a58e0b01048262fd
I20260926 03:37:04.981032 22308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:37:05.005055 22308 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:37:05.006546 22308 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:37:05.017649 22308 catalog_manager.cc:5868] T 00000000000000000000000000000000 P daffe8af83c140cc8a3fc401563316e3: Generated new TSK 0
I20260926 03:37:05.018998 22308 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:37:05.026216 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 22225
W20260926 03:37:05.027633 22287 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57033
W20260926 03:37:05.028636 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 22225: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:37:05.028803 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 22225
W20260926 03:37:05.048651 22210 connection.cc:535] client connection to 127.0.97.62:42921 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:42921): Connection reset by peer (error 104)
2026-09-26T03:37:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2080 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:37:05 dist-test-slave-dt7p krb5kdc[22323](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:37:05 dist-test-slave-dt7p krb5kdc[22323](info): set up 2 sockets
Sep 26 03:37:05 dist-test-slave-dt7p krb5kdc[22323](info): commencing operation
krb5kdc: starting...
W20260926 03:37:06.165508 388 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.006s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:37:06 dist-test-slave-dt7p krb5kdc[22323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:37:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:37:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:37:06.484429 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37485
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:37:06.884958 22336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:37:06.885326 22336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:37:06.885530 22336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:37:06.894327 22336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:37:06.894475 22336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:37:06.894587 22336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:37:06.894683 22336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:37:06.894754 22336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:37:06.906448 22336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37485
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:37:06.908246 22336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:37:06.909952 22336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:37:06.923525 22341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:06.923480 22344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:06.923550 22342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:06.924382 22336 server_base.cc:1034] running on GCE node
I20260926 03:37:06.925426 22336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:37:06.927109 22336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:37:06.928349 22336 hybrid_clock.cc:648] HybridClock initialized: now 1790393826928303 us; error 35 us; skew 500 ppm
I20260926 03:37:06.928818 22336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:37:06 dist-test-slave-dt7p krb5kdc[22323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:37:06.933056 22336 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:37:06.935052 22336 webserver.cc:460] Webserver started at http://127.0.97.62:38277/ using document root <none> and password file <none>
I20260926 03:37:06.935911 22336 fs_manager.cc:359] Metadata directory not provided
I20260926 03:37:06.936028 22336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:37:06.936353 22336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:37:06.940035 22336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "bcbeff73a288494aa11b1526df297729"
format_stamp: "Formatted at 2026-09-26 03:37:06 on dist-test-slave-dt7p"
server_key: "17390a1c014c5c9d00b7aafa0ec4a4d2"
server_key_iv: "2c36ef79be694f20c45d0e693c2e8eb3"
server_key_version: "encryptionkey@0"
I20260926 03:37:06.941002 22336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "bcbeff73a288494aa11b1526df297729"
format_stamp: "Formatted at 2026-09-26 03:37:06 on dist-test-slave-dt7p"
server_key: "17390a1c014c5c9d00b7aafa0ec4a4d2"
server_key_iv: "2c36ef79be694f20c45d0e693c2e8eb3"
server_key_version: "encryptionkey@0"
I20260926 03:37:06.947412 22336 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 03:37:06.952001 22351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:06.953697 22336 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 03:37:06.953871 22336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "bcbeff73a288494aa11b1526df297729"
format_stamp: "Formatted at 2026-09-26 03:37:06 on dist-test-slave-dt7p"
server_key: "17390a1c014c5c9d00b7aafa0ec4a4d2"
server_key_iv: "2c36ef79be694f20c45d0e693c2e8eb3"
server_key_version: "encryptionkey@0"
I20260926 03:37:06.954073 22336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:37:06.960896 22336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:37:06.965782 22336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:37:06.966063 22336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:37:06.987381 22336 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37485
I20260926 03:37:06.987419 22402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37485 every 8 connection(s)
I20260926 03:37:06.989117 22336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:37:06.993584 22403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:06.996750 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 22336
I20260926 03:37:06.996964 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:37:06.997627 388 external_mini_cluster.cc:1428] Setting key 3d1320362b6676b72a9d80d024ee8ef8
I20260926 03:37:07.006518 22406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:06.999713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40731 (local address 127.0.97.62:37485)
0926 03:37:07.000706 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:07.000720 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:07.000768 (+ 48us) server_negotiation.cc:407] Connection header received
0926 03:37:07.002570 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:07.002605 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:07.003057 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:07.003448 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:07.004704 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:07.004710 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:07.004727 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:37:07.004745 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:07.004835 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:37:07.004858 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:07.005054 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:07.005208 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 03:37:07.005467 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":448,"threads_started":1}
I20260926 03:37:07.007597 22403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcbeff73a288494aa11b1526df297729: Bootstrap starting.
I20260926 03:37:07.010555 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:37:07.011519 22403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcbeff73a288494aa11b1526df297729: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:07.012702 22403 log.cc:824] T 00000000000000000000000000000000 P bcbeff73a288494aa11b1526df297729: Log is configured to *not* fsync() on all Append() calls
I20260926 03:37:07.016359 22403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcbeff73a288494aa11b1526df297729: No bootstrap required, opened a new log
I20260926 03:37:07.018285 22406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:07.015492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40993 (local address 127.0.97.62:37485)
0926 03:37:07.015705 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:07.015713 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:07.016112 (+ 399us) server_negotiation.cc:407] Connection header received
0926 03:37:07.016303 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:07.016309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:07.016371 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:07.016486 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:07.017541 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:07.017545 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:07.017549 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:37:07.017563 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:07.017608 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:37:07.017624 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:07.017712 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:07.017856 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 03:37:07.017906 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 03:37:07.025718 22403 raft_consensus.cc:348] T 00000000000000000000000000000000 P bcbeff73a288494aa11b1526df297729 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcbeff73a288494aa11b1526df297729" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37485 } }
I20260926 03:37:07.026160 22403 raft_consensus.cc:374] T 00000000000000000000000000000000 P bcbeff73a288494aa11b1526df297729 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:07.026275 22403 raft_consensus.cc:729] T 00000000000000000000000000000000 P bcbeff73a288494aa11b1526df297729 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcbeff73a288494aa11b1526df297729, State: Initialized, Role: FOLLOWER
I20260926 03:37:07.026902 22403 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcbeff73a288494aa11b1526df297729 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcbeff73a288494aa11b1526df297729" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37485 } }
I20260926 03:37:07.027161 22403 raft_consensus.cc:388] T 00000000000000000000000000000000 P bcbeff73a288494aa11b1526df297729 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:37:07.027315 22403 raft_consensus.cc:482] T 00000000000000000000000000000000 P bcbeff73a288494aa11b1526df297729 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:37:07.027493 22403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bcbeff73a288494aa11b1526df297729 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:07.030243 22403 raft_consensus.cc:504] T 00000000000000000000000000000000 P bcbeff73a288494aa11b1526df297729 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcbeff73a288494aa11b1526df297729" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37485 } }
I20260926 03:37:07.030820 22403 leader_election.cc:302] T 00000000000000000000000000000000 P bcbeff73a288494aa11b1526df297729 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bcbeff73a288494aa11b1526df297729; no voters:
I20260926 03:37:07.031492 22403 leader_election.cc:288] T 00000000000000000000000000000000 P bcbeff73a288494aa11b1526df297729 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:37:07.031647 22413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bcbeff73a288494aa11b1526df297729 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:37:07.032387 22413 raft_consensus.cc:686] T 00000000000000000000000000000000 P bcbeff73a288494aa11b1526df297729 [term 1 LEADER]: Becoming Leader. State: Replica: bcbeff73a288494aa11b1526df297729, State: Running, Role: LEADER
I20260926 03:37:07.033020 22413 consensus_queue.cc:237] T 00000000000000000000000000000000 P bcbeff73a288494aa11b1526df297729 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcbeff73a288494aa11b1526df297729" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37485 } }
I20260926 03:37:07.033918 22403 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcbeff73a288494aa11b1526df297729 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:37:07.036795 22414 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcbeff73a288494aa11b1526df297729 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bcbeff73a288494aa11b1526df297729" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcbeff73a288494aa11b1526df297729" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37485 } } }
I20260926 03:37:07.036835 22415 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcbeff73a288494aa11b1526df297729 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcbeff73a288494aa11b1526df297729. Latest consensus state: current_term: 1 leader_uuid: "bcbeff73a288494aa11b1526df297729" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcbeff73a288494aa11b1526df297729" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37485 } } }
I20260926 03:37:07.037262 22414 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcbeff73a288494aa11b1526df297729 [sys.catalog]: This master's current role is: LEADER
I20260926 03:37:07.037283 22415 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcbeff73a288494aa11b1526df297729 [sys.catalog]: This master's current role is: LEADER
I20260926 03:37:07.039608 22419 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:37:07.045563 22419 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:37:07.056035 22419 catalog_manager.cc:1347] Generated new cluster ID: 07f9a1c57f9340a688ce99b4a67d86e8
I20260926 03:37:07.056159 22419 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:37:07.065487 22419 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:37:07.065964 22419 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:37:07.073339 22419 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bcbeff73a288494aa11b1526df297729: Generated new TSK 0
I20260926 03:37:07.074251 22419 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:37:07.083647 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 22336
W20260926 03:37:07.084949 22398 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40731
W20260926 03:37:07.085839 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 22336: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:37:07.085990 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 22336
2026-09-26T03:37:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2056 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:37:07 dist-test-slave-dt7p krb5kdc[22434](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:37:07 dist-test-slave-dt7p krb5kdc[22434](info): set up 2 sockets
Sep 26 03:37:07 dist-test-slave-dt7p krb5kdc[22434](info): commencing operation
krb5kdc: starting...
W20260926 03:37:08.217497 388 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.001s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:37:08 dist-test-slave-dt7p krb5kdc[22434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:37:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:37:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:37:08.512106 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33087
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:37:08.899183 22447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:37:08.899511 22447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:37:08.899719 22447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:37:08.907310 22447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:37:08.907433 22447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:37:08.907545 22447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:37:08.907665 22447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:37:08.907749 22447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:37:08.917867 22447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:37:08.919461 22447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:37:08.921095 22447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:37:08.934394 22452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:08.934461 22455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:08.934468 22453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:08.935613 22447 server_base.cc:1034] running on GCE node
I20260926 03:37:08.936436 22447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:37:08.938086 22447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:37:08.939316 22447 hybrid_clock.cc:648] HybridClock initialized: now 1790393828939273 us; error 26 us; skew 500 ppm
I20260926 03:37:08.939754 22447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:37:08 dist-test-slave-dt7p krb5kdc[22434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:37:08.949536 22447 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:37:08.951393 22447 webserver.cc:460] Webserver started at http://127.0.97.62:39637/ using document root <none> and password file <none>
I20260926 03:37:08.952282 22447 fs_manager.cc:359] Metadata directory not provided
I20260926 03:37:08.952440 22447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:37:08.952858 22447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:37:08.956292 22447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "8a87c8c1a87747139266d637a397daaa"
format_stamp: "Formatted at 2026-09-26 03:37:08 on dist-test-slave-dt7p"
server_key: "590bc06eec34ad7d7323fdaaf0b62163"
server_key_iv: "b1fb84ee83082b6da7954b7bd787c0b0"
server_key_version: "encryptionkey@0"
I20260926 03:37:08.957216 22447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "8a87c8c1a87747139266d637a397daaa"
format_stamp: "Formatted at 2026-09-26 03:37:08 on dist-test-slave-dt7p"
server_key: "590bc06eec34ad7d7323fdaaf0b62163"
server_key_iv: "b1fb84ee83082b6da7954b7bd787c0b0"
server_key_version: "encryptionkey@0"
I20260926 03:37:08.963160 22447 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 03:37:08.967226 22462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:08.968984 22447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:37:08.969213 22447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "8a87c8c1a87747139266d637a397daaa"
format_stamp: "Formatted at 2026-09-26 03:37:08 on dist-test-slave-dt7p"
server_key: "590bc06eec34ad7d7323fdaaf0b62163"
server_key_iv: "b1fb84ee83082b6da7954b7bd787c0b0"
server_key_version: "encryptionkey@0"
I20260926 03:37:08.969462 22447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:37:08.985246 22447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:37:08.990332 22447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:37:08.990700 22447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:37:09.010358 22447 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33087
I20260926 03:37:09.010383 22513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33087 every 8 connection(s)
I20260926 03:37:09.012068 22447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:37:09.013195 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 22447
I20260926 03:37:09.013419 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:37:09.013916 388 external_mini_cluster.cc:1428] Setting key 7321ea44c61e87575909d780da9c0b49
I20260926 03:37:09.017151 22514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 03:37:09 dist-test-slave-dt7p krb5kdc[22434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:37:09.032235 22514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a87c8c1a87747139266d637a397daaa: Bootstrap starting.
I20260926 03:37:09.036664 22514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a87c8c1a87747139266d637a397daaa: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:09.038198 22514 log.cc:824] T 00000000000000000000000000000000 P 8a87c8c1a87747139266d637a397daaa: Log is configured to *not* fsync() on all Append() calls
I20260926 03:37:09.040961 22517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:09.016008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33721 (local address 127.0.97.62:33087)
0926 03:37:09.016961 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:09.016974 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:09.017023 (+ 49us) server_negotiation.cc:407] Connection header received
0926 03:37:09.018858 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:09.018883 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:09.019319 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:09.019699 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:09.021668 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:09.023138 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:09.025079 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:09.025809 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:09.025965 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:09.031684 (+ 5719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:09.031698 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:09.031722 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:09.031778 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:09.035699 (+ 3921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:09.036796 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:09.036807 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:09.036825 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:09.037053 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:09.037576 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:09.037580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:09.037583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:09.038213 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:09.038574 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:09.039070 (+ 496us) server_negotiation.cc:299] Negotiation successful
0926 03:37:09.039676 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":469,"threads_started":1}
I20260926 03:37:09.041909 22514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a87c8c1a87747139266d637a397daaa: No bootstrap required, opened a new log
I20260926 03:37:09.045341 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:37:09.051010 22514 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a87c8c1a87747139266d637a397daaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a87c8c1a87747139266d637a397daaa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33087 } }
I20260926 03:37:09.051628 22514 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a87c8c1a87747139266d637a397daaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:09.051771 22514 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a87c8c1a87747139266d637a397daaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a87c8c1a87747139266d637a397daaa, State: Initialized, Role: FOLLOWER
I20260926 03:37:09.052675 22514 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a87c8c1a87747139266d637a397daaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a87c8c1a87747139266d637a397daaa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33087 } }
I20260926 03:37:09.053005 22514 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a87c8c1a87747139266d637a397daaa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:37:09.053215 22514 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a87c8c1a87747139266d637a397daaa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:37:09.053419 22514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a87c8c1a87747139266d637a397daaa [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:09.056814 22514 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a87c8c1a87747139266d637a397daaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a87c8c1a87747139266d637a397daaa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33087 } }
I20260926 03:37:09.057502 22514 leader_election.cc:302] T 00000000000000000000000000000000 P 8a87c8c1a87747139266d637a397daaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a87c8c1a87747139266d637a397daaa; no voters:
I20260926 03:37:09.058264 22514 leader_election.cc:288] T 00000000000000000000000000000000 P 8a87c8c1a87747139266d637a397daaa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:37:09.058393 22524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a87c8c1a87747139266d637a397daaa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:37:09.059870 22524 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a87c8c1a87747139266d637a397daaa [term 1 LEADER]: Becoming Leader. State: Replica: 8a87c8c1a87747139266d637a397daaa, State: Running, Role: LEADER
I20260926 03:37:09.060714 22524 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a87c8c1a87747139266d637a397daaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a87c8c1a87747139266d637a397daaa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33087 } }
I20260926 03:37:09.061100 22514 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a87c8c1a87747139266d637a397daaa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:37:09.065310 22525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a87c8c1a87747139266d637a397daaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a87c8c1a87747139266d637a397daaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a87c8c1a87747139266d637a397daaa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33087 } } }
I20260926 03:37:09.065738 22525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a87c8c1a87747139266d637a397daaa [sys.catalog]: This master's current role is: LEADER
I20260926 03:37:09.066864 22532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:37:09.065269 22526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a87c8c1a87747139266d637a397daaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a87c8c1a87747139266d637a397daaa. Latest consensus state: current_term: 1 leader_uuid: "8a87c8c1a87747139266d637a397daaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a87c8c1a87747139266d637a397daaa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33087 } } }
I20260926 03:37:09.068344 22526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a87c8c1a87747139266d637a397daaa [sys.catalog]: This master's current role is: LEADER
I20260926 03:37:09.071254 22517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:09.050386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58195 (local address 127.0.97.62:33087)
0926 03:37:09.050627 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:09.050634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:09.051025 (+ 391us) server_negotiation.cc:407] Connection header received
0926 03:37:09.051268 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:09.051273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:09.051344 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:09.051434 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:09.053656 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:09.054483 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:09.056317 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:09.056880 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:09.057000 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:09.060263 (+ 3263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:09.060273 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:09.060280 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:09.060318 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:09.065463 (+ 5145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:09.066635 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:09.066641 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:09.066646 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:09.066875 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:09.067385 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:09.067391 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:09.067395 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:09.067823 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:09.068133 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:09.070256 (+ 2123us) server_negotiation.cc:299] Negotiation successful
0926 03:37:09.070619 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 03:37:09.078759 22532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:37:09.090595 22532 catalog_manager.cc:1347] Generated new cluster ID: b143361f5e824d13bc2b7b45c3f6ee3a
I20260926 03:37:09.090749 22532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:37:09.116387 22532 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:37:09.118292 22532 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:37:09.128748 22532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a87c8c1a87747139266d637a397daaa: Generated new TSK 0
I20260926 03:37:09.129984 22532 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:37:09.143615 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 22447
I20260926 03:37:09.145123 22509 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:37:09.248507 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 22447
2026-09-26T03:37:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2163 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:37:09 dist-test-slave-dt7p krb5kdc[22546](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:37:09 dist-test-slave-dt7p krb5kdc[22546](info): set up 2 sockets
Sep 26 03:37:09 dist-test-slave-dt7p krb5kdc[22546](info): commencing operation
krb5kdc: starting...
W20260926 03:37:10.381626 388 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.007s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:37:10 dist-test-slave-dt7p krb5kdc[22546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:37:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:37:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:37:10.693861 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45695
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:37:11.078898 22559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:37:11.079257 22559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:37:11.079493 22559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:37:11.087345 22559 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:37:11.087491 22559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:37:11.087626 22559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:37:11.087736 22559 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:37:11.087821 22559 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:37:11.100260 22559 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:37:11.101989 22559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:37:11.103628 22559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:37:11.117530 22565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:11.118005 22567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:11.117580 22564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:11.118522 22559 server_base.cc:1034] running on GCE node
I20260926 03:37:11.119408 22559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:37:11.121206 22559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:37:11.122454 22559 hybrid_clock.cc:648] HybridClock initialized: now 1790393831122416 us; error 58 us; skew 500 ppm
I20260926 03:37:11.122864 22559 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:37:11 dist-test-slave-dt7p krb5kdc[22546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:37:11.126577 22559 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:37:11.128305 22559 webserver.cc:460] Webserver started at http://127.0.97.62:41709/ using document root <none> and password file <none>
I20260926 03:37:11.129170 22559 fs_manager.cc:359] Metadata directory not provided
I20260926 03:37:11.129290 22559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:37:11.129585 22559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:37:11.133116 22559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "efa4722b02754f3e98d13da42bd72ede"
format_stamp: "Formatted at 2026-09-26 03:37:11 on dist-test-slave-dt7p"
server_key: "c99f6e3b3604bc717758cf372c303daf"
server_key_iv: "811013d95cf7e62559a330382d46dbb4"
server_key_version: "encryptionkey@0"
I20260926 03:37:11.133931 22559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "efa4722b02754f3e98d13da42bd72ede"
format_stamp: "Formatted at 2026-09-26 03:37:11 on dist-test-slave-dt7p"
server_key: "c99f6e3b3604bc717758cf372c303daf"
server_key_iv: "811013d95cf7e62559a330382d46dbb4"
server_key_version: "encryptionkey@0"
I20260926 03:37:11.140005 22559 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260926 03:37:11.144069 22574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:11.145740 22559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:37:11.145897 22559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "efa4722b02754f3e98d13da42bd72ede"
format_stamp: "Formatted at 2026-09-26 03:37:11 on dist-test-slave-dt7p"
server_key: "c99f6e3b3604bc717758cf372c303daf"
server_key_iv: "811013d95cf7e62559a330382d46dbb4"
server_key_version: "encryptionkey@0"
I20260926 03:37:11.146081 22559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:37:11.169502 22559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:37:11.174336 22559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:37:11.174618 22559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:37:11.195487 22559 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45695
I20260926 03:37:11.195511 22625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45695 every 8 connection(s)
I20260926 03:37:11.197202 22559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:37:11.201882 22626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:11.204514 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 22559
I20260926 03:37:11.204769 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:37:11.205372 388 external_mini_cluster.cc:1428] Setting key e3b544111c2e965b5d72e51d061a1785
I20260926 03:37:11.215672 22626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efa4722b02754f3e98d13da42bd72ede: Bootstrap starting.
I20260926 03:37:11.219650 22626 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efa4722b02754f3e98d13da42bd72ede: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:11.220609 22629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:11.207832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34343 (local address 127.0.97.62:45695)
0926 03:37:11.208823 (+ 991us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:11.208838 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:11.208894 (+ 56us) server_negotiation.cc:407] Connection header received
0926 03:37:11.210832 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:11.210862 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:11.211369 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:11.211829 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:11.212490 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:11.214030 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:11.215457 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:11.216263 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:11.216479 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:11.217906 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:11.217916 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:11.217931 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:37:11.218549 (+ 618us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:11.218637 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:37:11.218664 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:11.218803 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:11.219035 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 03:37:11.219332 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":403,"threads_started":1}
I20260926 03:37:11.221007 22626 log.cc:824] T 00000000000000000000000000000000 P efa4722b02754f3e98d13da42bd72ede: Log is configured to *not* fsync() on all Append() calls
I20260926 03:37:11.224440 22626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efa4722b02754f3e98d13da42bd72ede: No bootstrap required, opened a new log
I20260926 03:37:11.224844 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:37:11.232242 22626 raft_consensus.cc:348] T 00000000000000000000000000000000 P efa4722b02754f3e98d13da42bd72ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efa4722b02754f3e98d13da42bd72ede" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45695 } }
I20260926 03:37:11.232815 22626 raft_consensus.cc:374] T 00000000000000000000000000000000 P efa4722b02754f3e98d13da42bd72ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:11.232946 22626 raft_consensus.cc:729] T 00000000000000000000000000000000 P efa4722b02754f3e98d13da42bd72ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efa4722b02754f3e98d13da42bd72ede, State: Initialized, Role: FOLLOWER
I20260926 03:37:11.233685 22626 consensus_queue.cc:260] T 00000000000000000000000000000000 P efa4722b02754f3e98d13da42bd72ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efa4722b02754f3e98d13da42bd72ede" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45695 } }
I20260926 03:37:11.233946 22626 raft_consensus.cc:388] T 00000000000000000000000000000000 P efa4722b02754f3e98d13da42bd72ede [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:37:11.234107 22626 raft_consensus.cc:482] T 00000000000000000000000000000000 P efa4722b02754f3e98d13da42bd72ede [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:37:11.234302 22626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efa4722b02754f3e98d13da42bd72ede [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:11.237481 22626 raft_consensus.cc:504] T 00000000000000000000000000000000 P efa4722b02754f3e98d13da42bd72ede [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efa4722b02754f3e98d13da42bd72ede" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45695 } }
I20260926 03:37:11.238158 22626 leader_election.cc:302] T 00000000000000000000000000000000 P efa4722b02754f3e98d13da42bd72ede [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efa4722b02754f3e98d13da42bd72ede; no voters:
I20260926 03:37:11.238840 22626 leader_election.cc:288] T 00000000000000000000000000000000 P efa4722b02754f3e98d13da42bd72ede [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:37:11.238991 22636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efa4722b02754f3e98d13da42bd72ede [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:37:11.241020 22636 raft_consensus.cc:686] T 00000000000000000000000000000000 P efa4722b02754f3e98d13da42bd72ede [term 1 LEADER]: Becoming Leader. State: Replica: efa4722b02754f3e98d13da42bd72ede, State: Running, Role: LEADER
I20260926 03:37:11.241797 22636 consensus_queue.cc:237] T 00000000000000000000000000000000 P efa4722b02754f3e98d13da42bd72ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efa4722b02754f3e98d13da42bd72ede" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45695 } }
I20260926 03:37:11.242421 22626 sys_catalog.cc:564] T 00000000000000000000000000000000 P efa4722b02754f3e98d13da42bd72ede [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:37:11.243564 22629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:11.234851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35419 (local address 127.0.97.62:45695)
0926 03:37:11.235112 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:11.235122 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:11.235729 (+ 607us) server_negotiation.cc:407] Connection header received
0926 03:37:11.235866 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:11.235873 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:11.235969 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:11.236096 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:11.236789 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:11.237922 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:11.240042 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:11.240830 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:11.240959 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:11.242690 (+ 1731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:11.242697 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:11.242702 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:37:11.242727 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:11.242780 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:37:11.242804 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:11.242895 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:11.243083 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 03:37:11.243197 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 03:37:11.249832 22637 sys_catalog.cc:455] T 00000000000000000000000000000000 P efa4722b02754f3e98d13da42bd72ede [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "efa4722b02754f3e98d13da42bd72ede" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efa4722b02754f3e98d13da42bd72ede" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45695 } } }
I20260926 03:37:11.250264 22637 sys_catalog.cc:458] T 00000000000000000000000000000000 P efa4722b02754f3e98d13da42bd72ede [sys.catalog]: This master's current role is: LEADER
I20260926 03:37:11.251554 22638 sys_catalog.cc:455] T 00000000000000000000000000000000 P efa4722b02754f3e98d13da42bd72ede [sys.catalog]: SysCatalogTable state changed. Reason: New leader efa4722b02754f3e98d13da42bd72ede. Latest consensus state: current_term: 1 leader_uuid: "efa4722b02754f3e98d13da42bd72ede" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efa4722b02754f3e98d13da42bd72ede" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45695 } } }
I20260926 03:37:11.251988 22638 sys_catalog.cc:458] T 00000000000000000000000000000000 P efa4722b02754f3e98d13da42bd72ede [sys.catalog]: This master's current role is: LEADER
I20260926 03:37:11.252818 22646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:37:11.259689 22646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:37:11.271045 22646 catalog_manager.cc:1347] Generated new cluster ID: 8b93196234774e039e9dd1ff12706d34
I20260926 03:37:11.271191 22646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:37:11.295773 22646 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:37:11.297416 22646 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:37:11.311408 22646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efa4722b02754f3e98d13da42bd72ede: Generated new TSK 0
I20260926 03:37:11.312342 22646 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:37:11.323824 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 22559
W20260926 03:37:11.325407 22621 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34343
W20260926 03:37:11.326123 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 22559: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:37:11.326280 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 22559
2026-09-26T03:37:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2076 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:37:11 dist-test-slave-dt7p krb5kdc[22657](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:37:11 dist-test-slave-dt7p krb5kdc[22657](info): set up 2 sockets
Sep 26 03:37:11 dist-test-slave-dt7p krb5kdc[22657](info): commencing operation
krb5kdc: starting...
W20260926 03:37:12.457382 388 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.000s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:37:12 dist-test-slave-dt7p krb5kdc[22657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:37:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:37:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:37:12.758302 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34653
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:37:13.150273 22670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:37:13.150607 22670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:37:13.150780 22670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:37:13.158627 22670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:37:13.158757 22670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:37:13.158871 22670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:37:13.158983 22670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:37:13.159055 22670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:37:13.169842 22670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:37:13.171429 22670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:37:13.173019 22670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:37:13.187871 22675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:13.187947 22676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:13.188026 22670 server_base.cc:1034] running on GCE node
W20260926 03:37:13.188150 22678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:13.189278 22670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:37:13.190795 22670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:37:13.192044 22670 hybrid_clock.cc:648] HybridClock initialized: now 1790393833192002 us; error 46 us; skew 500 ppm
I20260926 03:37:13.192420 22670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:37:13 dist-test-slave-dt7p krb5kdc[22657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:37:13.195804 22670 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:37:13.197499 22670 webserver.cc:460] Webserver started at http://127.0.97.62:43189/ using document root <none> and password file <none>
I20260926 03:37:13.198324 22670 fs_manager.cc:359] Metadata directory not provided
I20260926 03:37:13.198473 22670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:37:13.198818 22670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:37:13.202241 22670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "6681a96ab3324a6b83f0e9c8a3c1e05c"
format_stamp: "Formatted at 2026-09-26 03:37:13 on dist-test-slave-dt7p"
server_key: "8f4f9db4b48be3d959c8d7c665ab9592"
server_key_iv: "6f0bad31f71b6ac0a005cb13f75423ec"
server_key_version: "encryptionkey@0"
I20260926 03:37:13.203094 22670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "6681a96ab3324a6b83f0e9c8a3c1e05c"
format_stamp: "Formatted at 2026-09-26 03:37:13 on dist-test-slave-dt7p"
server_key: "8f4f9db4b48be3d959c8d7c665ab9592"
server_key_iv: "6f0bad31f71b6ac0a005cb13f75423ec"
server_key_version: "encryptionkey@0"
I20260926 03:37:13.209107 22670 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 03:37:13.213029 22685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:13.214474 22670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:37:13.214679 22670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "6681a96ab3324a6b83f0e9c8a3c1e05c"
format_stamp: "Formatted at 2026-09-26 03:37:13 on dist-test-slave-dt7p"
server_key: "8f4f9db4b48be3d959c8d7c665ab9592"
server_key_iv: "6f0bad31f71b6ac0a005cb13f75423ec"
server_key_version: "encryptionkey@0"
I20260926 03:37:13.214910 22670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:37:13.221480 22670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:37:13.226225 22670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:37:13.226575 22670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:37:13.248090 22670 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34653
I20260926 03:37:13.248126 22736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34653 every 8 connection(s)
I20260926 03:37:13.249848 22670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:37:13.254436 22737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:13.259672 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 22670
I20260926 03:37:13.259889 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:37:13.260499 388 external_mini_cluster.cc:1428] Setting key a565b79e9ea1c9f373e2fdec4f81bfb8
I20260926 03:37:13.268450 22737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6681a96ab3324a6b83f0e9c8a3c1e05c: Bootstrap starting.
I20260926 03:37:13.270792 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:13.262805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60215 (local address 127.0.97.62:34653)
0926 03:37:13.263756 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:13.263770 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:13.263820 (+ 50us) server_negotiation.cc:407] Connection header received
0926 03:37:13.266524 (+ 2704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:13.266556 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:13.267035 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:13.267463 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:13.268857 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:13.268870 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:13.268886 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:37:13.268906 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:13.268994 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:37:13.269020 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:13.269253 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:13.269367 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 03:37:13.269659 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":431,"threads_started":1}
I20260926 03:37:13.273000 22737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6681a96ab3324a6b83f0e9c8a3c1e05c: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:13.274507 22737 log.cc:824] T 00000000000000000000000000000000 P 6681a96ab3324a6b83f0e9c8a3c1e05c: Log is configured to *not* fsync() on all Append() calls
I20260926 03:37:13.275430 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:37:13.278348 22737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6681a96ab3324a6b83f0e9c8a3c1e05c: No bootstrap required, opened a new log
I20260926 03:37:13.283985 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:13.281242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44615 (local address 127.0.97.62:34653)
0926 03:37:13.281505 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:13.281517 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:13.281855 (+ 338us) server_negotiation.cc:407] Connection header received
0926 03:37:13.282036 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:13.282040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:13.282095 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:13.282192 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:13.283260 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:13.283268 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:13.283272 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:37:13.283288 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:13.283334 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:37:13.283353 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:13.283449 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:13.283593 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 03:37:13.283658 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260926 03:37:13.288726 22737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6681a96ab3324a6b83f0e9c8a3c1e05c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6681a96ab3324a6b83f0e9c8a3c1e05c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34653 } }
I20260926 03:37:13.289155 22737 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6681a96ab3324a6b83f0e9c8a3c1e05c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:37:13.289284 22737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6681a96ab3324a6b83f0e9c8a3c1e05c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6681a96ab3324a6b83f0e9c8a3c1e05c, State: Initialized, Role: FOLLOWER
I20260926 03:37:13.289964 22737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6681a96ab3324a6b83f0e9c8a3c1e05c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6681a96ab3324a6b83f0e9c8a3c1e05c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34653 } }
I20260926 03:37:13.290246 22737 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6681a96ab3324a6b83f0e9c8a3c1e05c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:37:13.290409 22737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6681a96ab3324a6b83f0e9c8a3c1e05c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:37:13.290599 22737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6681a96ab3324a6b83f0e9c8a3c1e05c [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:13.293512 22737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6681a96ab3324a6b83f0e9c8a3c1e05c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6681a96ab3324a6b83f0e9c8a3c1e05c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34653 } }
I20260926 03:37:13.294121 22737 leader_election.cc:302] T 00000000000000000000000000000000 P 6681a96ab3324a6b83f0e9c8a3c1e05c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6681a96ab3324a6b83f0e9c8a3c1e05c; no voters:
I20260926 03:37:13.294764 22737 leader_election.cc:288] T 00000000000000000000000000000000 P 6681a96ab3324a6b83f0e9c8a3c1e05c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:37:13.294910 22747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6681a96ab3324a6b83f0e9c8a3c1e05c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:37:13.296451 22747 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6681a96ab3324a6b83f0e9c8a3c1e05c [term 1 LEADER]: Becoming Leader. State: Replica: 6681a96ab3324a6b83f0e9c8a3c1e05c, State: Running, Role: LEADER
I20260926 03:37:13.297039 22747 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6681a96ab3324a6b83f0e9c8a3c1e05c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6681a96ab3324a6b83f0e9c8a3c1e05c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34653 } }
I20260926 03:37:13.297425 22737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6681a96ab3324a6b83f0e9c8a3c1e05c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:37:13.299958 22749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6681a96ab3324a6b83f0e9c8a3c1e05c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6681a96ab3324a6b83f0e9c8a3c1e05c. Latest consensus state: current_term: 1 leader_uuid: "6681a96ab3324a6b83f0e9c8a3c1e05c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6681a96ab3324a6b83f0e9c8a3c1e05c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34653 } } }
I20260926 03:37:13.300352 22749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6681a96ab3324a6b83f0e9c8a3c1e05c [sys.catalog]: This master's current role is: LEADER
I20260926 03:37:13.300551 22748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6681a96ab3324a6b83f0e9c8a3c1e05c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6681a96ab3324a6b83f0e9c8a3c1e05c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6681a96ab3324a6b83f0e9c8a3c1e05c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34653 } } }
I20260926 03:37:13.300951 22748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6681a96ab3324a6b83f0e9c8a3c1e05c [sys.catalog]: This master's current role is: LEADER
I20260926 03:37:13.309284 22753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:37:13.315483 22753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:37:13.325542 22753 catalog_manager.cc:1347] Generated new cluster ID: 13fa4492ab504ae4a744cb384cf43601
I20260926 03:37:13.325711 22753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:37:13.355973 22753 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:37:13.356669 22753 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:37:13.370820 22753 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6681a96ab3324a6b83f0e9c8a3c1e05c: Generated new TSK 0
I20260926 03:37:13.371865 22753 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:37:13.389438 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 22670
W20260926 03:37:13.390614 22732 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60215
W20260926 03:37:13.391420 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 22670: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:37:13.391587 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 22670
2026-09-26T03:37:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2066 ms)
[----------] 18 tests from AuthTokenIssuingTest (37623 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:37:13 dist-test-slave-dt7p krb5kdc[22769](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:37:13 dist-test-slave-dt7p krb5kdc[22769](info): set up 2 sockets
Sep 26 03:37:13 dist-test-slave-dt7p krb5kdc[22769](info): commencing operation
krb5kdc: starting...
W20260926 03:37:14.541703 388 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.012s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:37:14 dist-test-slave-dt7p krb5kdc[22769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:37:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:37:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:37:14.848340 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39963
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39963,127.0.97.61:44135,127.0.97.60:42499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:37:15.252456 22782 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 03:37:15.253019 22782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:37:15.253119 22782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:37:15.253314 22782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:37:15.261821 22782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:37:15.261937 22782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:37:15.262048 22782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:37:15.262145 22782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:37:15.262276 22782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:37:15.273855 22782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39963,127.0.97.61:44135,127.0.97.60:42499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:37:15.275589 22782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:37:15.277290 22782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:37:15.291368 22790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:15.291394 22788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:15.291478 22787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:15.292186 22782 server_base.cc:1034] running on GCE node
I20260926 03:37:15.293170 22782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:37:15.294858 22782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:37:15.296101 22782 hybrid_clock.cc:648] HybridClock initialized: now 1790393835296039 us; error 31 us; skew 500 ppm
I20260926 03:37:15.296535 22782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:37:15 dist-test-slave-dt7p krb5kdc[22769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:37:15.300176 22782 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:37:15.301963 22782 webserver.cc:460] Webserver started at http://127.0.97.62:35397/ using document root <none> and password file <none>
I20260926 03:37:15.302825 22782 fs_manager.cc:359] Metadata directory not provided
I20260926 03:37:15.302970 22782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:37:15.303349 22782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:37:15.306900 22782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "786111c01d7f4c91abdcff6155c42ec5"
format_stamp: "Formatted at 2026-09-26 03:37:15 on dist-test-slave-dt7p"
server_key: "6a53d17d25b10bc521bf5a2662e93996"
server_key_iv: "dc202fd90d836d5c1bc029f0fdbe9e61"
server_key_version: "encryptionkey@0"
I20260926 03:37:15.307789 22782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "786111c01d7f4c91abdcff6155c42ec5"
format_stamp: "Formatted at 2026-09-26 03:37:15 on dist-test-slave-dt7p"
server_key: "6a53d17d25b10bc521bf5a2662e93996"
server_key_iv: "dc202fd90d836d5c1bc029f0fdbe9e61"
server_key_version: "encryptionkey@0"
I20260926 03:37:15.313656 22782 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 03:37:15.317657 22797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:15.319279 22782 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:37:15.319497 22782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "786111c01d7f4c91abdcff6155c42ec5"
format_stamp: "Formatted at 2026-09-26 03:37:15 on dist-test-slave-dt7p"
server_key: "6a53d17d25b10bc521bf5a2662e93996"
server_key_iv: "dc202fd90d836d5c1bc029f0fdbe9e61"
server_key_version: "encryptionkey@0"
I20260926 03:37:15.319756 22782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:37:15.332223 22782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:37:15.336902 22782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:37:15.337224 22782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:37:15.359340 22782 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39963
I20260926 03:37:15.359398 22848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39963 every 8 connection(s)
I20260926 03:37:15.361133 22782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:37:15.366571 22849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:15.371567 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 22782
I20260926 03:37:15.371891 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:37:15.372664 388 external_mini_cluster.cc:1428] Setting key 4079fb570f9b21ef0b95700c48c313bc
I20260926 03:37:15.374866 22849 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:37:15 dist-test-slave-dt7p krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
I20260926 03:37:15.401578 22851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:15.378925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35396 (local address 127.0.97.62:39963)
0926 03:37:15.379910 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:15.379922 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:15.380596 (+ 674us) server_negotiation.cc:407] Connection header received
0926 03:37:15.383016 (+ 2420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:15.383022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:15.383308 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:15.383543 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:15.385217 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:15.386664 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:15.388200 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:15.388917 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:15.389139 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:15.393128 (+ 3989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:15.393138 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:15.393156 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:15.393188 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:15.396336 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:15.397172 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:15.397186 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:15.397188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:15.397348 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:15.397798 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:15.397803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:15.397805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:15.398427 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:15.398670 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:15.399261 (+ 591us) server_negotiation.cc:299] Negotiation successful
0926 03:37:15.399758 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":761,"thread_start_us":483,"threads_started":1}
I20260926 03:37:15.401597 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:15.378925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.0.1:35396)
0926 03:37:15.380376 (+ 1451us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:15.380422 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:15.382573 (+ 2151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:15.383567 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:15.383574 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:15.384448 (+ 874us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:15.384966 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:15.384988 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:15.386853 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:15.386858 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:15.388047 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:15.388058 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:15.389197 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:15.389203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:15.389371 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:15.390197 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:15.390223 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:15.392847 (+ 2624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:15.396497 (+ 3650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:15.396511 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:15.396527 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:15.397023 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:15.397478 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:15.397483 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:15.397486 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:15.397666 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:15.398574 (+ 908us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:15.398587 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:15.398840 (+ 253us) client_negotiation.cc:769] Sending connection context
0926 03:37:15.399121 (+ 281us) client_negotiation.cc:240] Negotiation successful
0926 03:37:15.399885 (+ 764us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1229,"mutex_wait_us":227,"thread_start_us":1025,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260926 03:37:15.405485 22849 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44135 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:37:15.406908 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:15.406417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44135 (local address 127.0.0.1:33076)
0926 03:37:15.406626 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:15.406764 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260926 03:37:15.407239 22798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44135: connect: Connection refused (error 111)
I20260926 03:37:15.409077 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:15.408682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44135 (local address 127.0.0.1:33088)
0926 03:37:15.408843 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:15.408930 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 03:37:15.409298 22849 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44135: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44135: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260926 03:37:15.444509 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:44135
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39963,127.0.97.61:44135,127.0.97.60:42499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:37:15.458717 22849 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44135 } attempt: 1
I20260926 03:37:15.460592 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:15.459843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44135 (local address 127.0.0.1:33094)
0926 03:37:15.460186 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:15.460347 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260926 03:37:15.461872 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:15.461395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44135 (local address 127.0.0.1:33102)
0926 03:37:15.461574 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:15.461673 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 03:37:15.462108 22849 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44135: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44135: connect: Connection refused (error 111)
I20260926 03:37:15.530402 22849 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44135 } attempt: 2
I20260926 03:37:15.532069 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:15.531342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44135 (local address 127.0.0.1:33116)
0926 03:37:15.531665 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:15.531825 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 03:37:15.533372 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:15.532895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44135 (local address 127.0.0.1:33126)
0926 03:37:15.533071 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:15.533183 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 03:37:15.533605 22849 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44135: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44135: connect: Connection refused (error 111)
I20260926 03:37:15.624929 22849 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44135 } attempt: 3
I20260926 03:37:15.626586 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:15.625868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44135 (local address 127.0.0.1:33142)
0926 03:37:15.626162 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:15.626329 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260926 03:37:15.627990 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:15.627445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44135 (local address 127.0.0.1:33144)
0926 03:37:15.627648 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:15.627759 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 03:37:15.628234 22849 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44135: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44135: connect: Connection refused (error 111)
I20260926 03:37:15.771590 22849 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44135 } attempt: 4
I20260926 03:37:15.773437 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:15.772644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44135 (local address 127.0.0.1:33158)
0926 03:37:15.773047 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:15.773190 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203}
I20260926 03:37:15.774852 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:15.774376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44135 (local address 127.0.0.1:33168)
0926 03:37:15.774556 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:15.774667 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 03:37:15.775084 22849 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44135: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44135: connect: Connection refused (error 111)
W20260926 03:37:15.837507 22856 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 03:37:15.837898 22856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:37:15.837968 22856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:37:15.838060 22856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:37:15.845899 22856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:37:15.846014 22856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:37:15.846079 22856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:37:15.846174 22856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:37:15.846243 22856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:37:15.857041 22856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39963,127.0.97.61:44135,127.0.97.60:42499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:44135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:37:15.858594 22856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:37:15.860067 22856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:37:15.873106 22864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:15.873186 22861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:15.873158 22862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:15.874389 22856 server_base.cc:1034] running on GCE node
I20260926 03:37:15.875198 22856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:37:15.876827 22856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:37:15.878065 22856 hybrid_clock.cc:648] HybridClock initialized: now 1790393835877971 us; error 80 us; skew 500 ppm
I20260926 03:37:15.878439 22856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:37:15 dist-test-slave-dt7p krb5kdc[22769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:37:15.881964 22856 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260926 03:37:15.883507 22856 webserver.cc:460] Webserver started at http://127.0.97.61:36601/ using document root <none> and password file <none>
I20260926 03:37:15.884284 22856 fs_manager.cc:359] Metadata directory not provided
I20260926 03:37:15.884405 22856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:37:15.884852 22856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:37:15.888162 22856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-1/data/instance:
uuid: "ae05067a21c644028932cc74e390c913"
format_stamp: "Formatted at 2026-09-26 03:37:15 on dist-test-slave-dt7p"
server_key: "69d73211bb104a493a6a72a4850d9c35"
server_key_iv: "2e5291751ca288a2cc270b194468421b"
server_key_version: "encryptionkey@0"
I20260926 03:37:15.889073 22856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-1/wal/instance:
uuid: "ae05067a21c644028932cc74e390c913"
format_stamp: "Formatted at 2026-09-26 03:37:15 on dist-test-slave-dt7p"
server_key: "69d73211bb104a493a6a72a4850d9c35"
server_key_iv: "2e5291751ca288a2cc270b194468421b"
server_key_version: "encryptionkey@0"
I20260926 03:37:15.894776 22856 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 03:37:15.898537 22871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:15.900081 22856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:37:15.900279 22856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-1/wal
uuid: "ae05067a21c644028932cc74e390c913"
format_stamp: "Formatted at 2026-09-26 03:37:15 on dist-test-slave-dt7p"
server_key: "69d73211bb104a493a6a72a4850d9c35"
server_key_iv: "2e5291751ca288a2cc270b194468421b"
server_key_version: "encryptionkey@0"
I20260926 03:37:15.900520 22856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:37:15.920292 22856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:37:15.924996 22856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:37:15.925305 22856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:37:15.946481 22856 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:44135
I20260926 03:37:15.946503 22922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:44135 every 8 connection(s)
I20260926 03:37:15.948097 22856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-1/data/info.pb
I20260926 03:37:15.953017 22923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:15.958439 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 22856
I20260926 03:37:15.958670 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-1/wal/instance
I20260926 03:37:15.959282 388 external_mini_cluster.cc:1428] Setting key 43fd183b913a60631040588eaf27b61f
I20260926 03:37:15.961359 22923 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:37:15 dist-test-slave-dt7p krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:37:15.988477 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:15.965372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35410 (local address 127.0.97.62:39963)
0926 03:37:15.966033 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:15.966043 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:15.967172 (+ 1129us) server_negotiation.cc:407] Connection header received
0926 03:37:15.969512 (+ 2340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:15.969522 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:15.969639 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:15.969798 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:15.971686 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:15.972934 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:15.975624 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:15.976060 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:15.976167 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:15.981271 (+ 5104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:15.981277 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:15.981281 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:15.981334 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:15.984419 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:15.985400 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:15.985408 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:15.985416 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:15.985557 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:15.986045 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:15.986056 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:15.986058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:15.986414 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:15.986598 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:15.987583 (+ 985us) server_negotiation.cc:299] Negotiation successful
0926 03:37:15.987894 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":324,"threads_started":1}
I20260926 03:37:15.989382 22926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:15.966005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.0.1:35410)
0926 03:37:15.966945 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:15.966996 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:15.969170 (+ 2174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:15.970023 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:15.970034 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:15.970678 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:15.971400 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:15.971420 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:15.973137 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:15.973146 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:15.975441 (+ 2295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:15.975459 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:15.976338 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:15.976347 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:15.976755 (+ 408us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:15.977729 (+ 974us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:15.977765 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:15.980952 (+ 3187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:15.984644 (+ 3692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:15.984656 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:15.984686 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:15.985241 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:15.985690 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:15.985696 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:15.985700 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:15.985915 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:15.986568 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:15.986578 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:15.987046 (+ 468us) client_negotiation.cc:769] Sending connection context
0926 03:37:15.987389 (+ 343us) client_negotiation.cc:240] Negotiation successful
0926 03:37:15.987923 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":413,"threads_started":1}
I20260926 03:37:15.991604 22923 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44135 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 26 03:37:15 dist-test-slave-dt7p krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 03:37:16.008068 22926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:15.992737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44135 (local address 127.0.0.1:33174)
0926 03:37:15.992943 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:15.992969 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:15.993094 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:15.994454 (+ 1360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:15.994458 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:15.994884 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:15.995112 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:15.995123 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:15.996680 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:15.996688 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:15.997555 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:15.997565 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:15.998443 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:15.998448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:15.998638 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:15.999321 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:15.999339 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:16.001600 (+ 2261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:16.005269 (+ 3669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:16.005278 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:16.005282 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:16.005665 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:16.006053 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:16.006060 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:16.006064 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:16.006206 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:16.006885 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:16.006890 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:16.007088 (+ 198us) client_negotiation.cc:769] Sending connection context
0926 03:37:16.007279 (+ 191us) client_negotiation.cc:240] Negotiation successful
0926 03:37:16.007598 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 03:37:16.008181 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:15.993078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33174 (local address 127.0.97.61:44135)
0926 03:37:15.993931 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:15.993938 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:15.993967 (+ 29us) server_negotiation.cc:407] Connection header received
0926 03:37:15.994014 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:15.994021 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:15.994228 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:15.994460 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:15.995321 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:15.996482 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:15.997706 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:15.998294 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:15.998436 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:16.001835 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:16.001846 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:16.001862 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:16.001891 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:16.005130 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:16.005808 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:16.005823 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:16.005825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:16.005939 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:16.006313 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:16.006319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:16.006324 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:16.006719 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:16.006967 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:16.007430 (+ 463us) server_negotiation.cc:299] Negotiation successful
0926 03:37:16.007725 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":667,"thread_start_us":510,"threads_started":1}
I20260926 03:37:16.011648 22923 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42499 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:37:16.013154 22926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:16.012624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42499 (local address 127.0.0.1:49700)
0926 03:37:16.012832 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:16.012960 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 03:37:16.013478 22874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42499: connect: Connection refused (error 111)
I20260926 03:37:16.015347 22926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:16.014967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42499 (local address 127.0.0.1:49710)
0926 03:37:16.015134 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:16.015220 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 03:37:16.015573 22923 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42499: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42499: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260926 03:37:16.033003 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:42499
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39963,127.0.97.61:44135,127.0.97.60:42499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:37:16.064936 22923 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42499 } attempt: 1
I20260926 03:37:16.066447 22926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:16.065860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42499 (local address 127.0.0.1:49724)
0926 03:37:16.066094 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:16.066245 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 03:37:16.067603 22926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:16.067254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42499 (local address 127.0.0.1:49732)
0926 03:37:16.067398 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:16.067493 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 03:37:16.067837 22923 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42499: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42499: connect: Connection refused (error 111)
I20260926 03:37:16.074358 22849 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44135 } attempt: 5
Sep 26 03:37:16 dist-test-slave-dt7p krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 03:37:16.090631 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:16.075178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44135 (local address 127.0.0.1:33176)
0926 03:37:16.075421 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:16.075448 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:16.075673 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:16.076112 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:16.076117 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:16.076544 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:16.076858 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:16.076869 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:16.078291 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:16.078295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:16.079300 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:16.079313 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:16.080286 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:16.080291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:16.080504 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:16.081349 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:16.081372 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:16.083764 (+ 2392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:16.087704 (+ 3940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:16.087711 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:16.087715 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:16.088177 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:16.088655 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:16.088665 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:16.088668 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:16.088821 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:16.089453 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:16.089458 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:16.089631 (+ 173us) client_negotiation.cc:769] Sending connection context
0926 03:37:16.089818 (+ 187us) client_negotiation.cc:240] Negotiation successful
0926 03:37:16.090198 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 03:37:16.090623 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:16.075318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33176 (local address 127.0.97.61:44135)
0926 03:37:16.075530 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:16.075536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:16.075562 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:37:16.075829 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:16.075835 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:16.075926 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:16.076042 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:16.077019 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:16.078114 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:16.079449 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:16.080130 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:16.080257 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:16.083978 (+ 3721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:16.083988 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:16.083993 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:16.084037 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:16.087533 (+ 3496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:16.088322 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:16.088330 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:16.088334 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:16.088488 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:16.088947 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:16.088955 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:16.088959 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:16.089314 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:16.089501 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:16.089946 (+ 445us) server_negotiation.cc:299] Negotiation successful
0926 03:37:16.090199 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 03:37:16.092056 22849 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42499 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:37:16.093305 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:16.092860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42499 (local address 127.0.0.1:49744)
0926 03:37:16.093047 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:16.093148 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 03:37:16.094264 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:16.093929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42499 (local address 127.0.0.1:49758)
0926 03:37:16.094074 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:16.094168 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 03:37:16.094473 22849 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42499: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42499: connect: Connection refused (error 111)
I20260926 03:37:16.136065 22923 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42499 } attempt: 2
I20260926 03:37:16.137667 22926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:16.137081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42499 (local address 127.0.0.1:49774)
0926 03:37:16.137359 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:16.137473 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260926 03:37:16.138760 22926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:16.138399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42499 (local address 127.0.0.1:49776)
0926 03:37:16.138552 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:16.138634 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 03:37:16.138999 22923 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42499: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42499: connect: Connection refused (error 111)
I20260926 03:37:16.145728 22849 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42499 } attempt: 1
I20260926 03:37:16.147061 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:16.146490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42499 (local address 127.0.0.1:49792)
0926 03:37:16.146708 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:16.146815 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 03:37:16.148214 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:16.147899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42499 (local address 127.0.0.1:49794)
0926 03:37:16.148033 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:16.148106 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 03:37:16.148488 22849 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42499: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42499: connect: Connection refused (error 111)
I20260926 03:37:16.216789 22849 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42499 } attempt: 2
I20260926 03:37:16.218596 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:16.217873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42499 (local address 127.0.0.1:49796)
0926 03:37:16.218185 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:16.218305 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260926 03:37:16.219878 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:16.219449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42499 (local address 127.0.0.1:49798)
0926 03:37:16.219646 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:16.219741 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 03:37:16.220160 22849 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42499: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42499: connect: Connection refused (error 111)
I20260926 03:37:16.230283 22923 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42499 } attempt: 3
I20260926 03:37:16.231760 22926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:16.231183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42499 (local address 127.0.0.1:49800)
0926 03:37:16.231447 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:16.231560 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 03:37:16.233151 22926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:16.232670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42499 (local address 127.0.0.1:49810)
0926 03:37:16.232930 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:16.233020 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
W20260926 03:37:16.233403 22923 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42499: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42499: connect: Connection refused (error 111)
I20260926 03:37:16.326457 22849 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42499 } attempt: 3
I20260926 03:37:16.328059 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:16.327329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42499 (local address 127.0.0.1:49820)
0926 03:37:16.327635 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:16.327782 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260926 03:37:16.329466 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:16.328914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42499 (local address 127.0.0.1:49824)
0926 03:37:16.329178 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:16.329292 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260926 03:37:16.329761 22849 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42499: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42499: connect: Connection refused (error 111)
I20260926 03:37:16.376722 22923 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42499 } attempt: 4
I20260926 03:37:16.378284 22926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:16.377699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42499 (local address 127.0.0.1:49826)
0926 03:37:16.377987 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:16.378092 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260926 03:37:16.379336 22926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:16.379012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42499 (local address 127.0.0.1:49828)
0926 03:37:16.379157 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:16.379232 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 03:37:16.379609 22923 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42499: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42499: connect: Connection refused (error 111)
W20260926 03:37:16.432238 22931 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 03:37:16.432801 22931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:37:16.432936 22931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:37:16.433105 22931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:37:16.441102 22931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:37:16.441216 22931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:37:16.441321 22931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:37:16.441427 22931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:37:16.441552 22931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:37:16.452960 22931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39963,127.0.97.61:44135,127.0.97.60:42499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:42499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:37:16.454692 22931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:37:16.456382 22931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:37:16.469761 22936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:16.469709 22937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:16.469792 22939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:16.471367 22931 server_base.cc:1034] running on GCE node
I20260926 03:37:16.472296 22931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:37:16.473917 22931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:37:16.475136 22931 hybrid_clock.cc:648] HybridClock initialized: now 1790393836475060 us; error 51 us; skew 500 ppm
I20260926 03:37:16.475526 22931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:37:16 dist-test-slave-dt7p krb5kdc[22769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:37:16.478962 22931 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260926 03:37:16.480731 22931 webserver.cc:460] Webserver started at http://127.0.97.60:45575/ using document root <none> and password file <none>
I20260926 03:37:16.481555 22931 fs_manager.cc:359] Metadata directory not provided
I20260926 03:37:16.481693 22931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:37:16.482061 22931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:37:16.485539 22931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-2/data/instance:
uuid: "8709711448fe472c83f8a559adcf2424"
format_stamp: "Formatted at 2026-09-26 03:37:16 on dist-test-slave-dt7p"
server_key: "15eb480e6ce1d7b622e975af0f39e9c1"
server_key_iv: "decec8a85972c5bed23d50387e659103"
server_key_version: "encryptionkey@0"
I20260926 03:37:16.486405 22931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-2/wal/instance:
uuid: "8709711448fe472c83f8a559adcf2424"
format_stamp: "Formatted at 2026-09-26 03:37:16 on dist-test-slave-dt7p"
server_key: "15eb480e6ce1d7b622e975af0f39e9c1"
server_key_iv: "decec8a85972c5bed23d50387e659103"
server_key_version: "encryptionkey@0"
I20260926 03:37:16.492079 22931 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:37:16.495910 22946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:16.497690 22931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:37:16.497843 22931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-2/wal
uuid: "8709711448fe472c83f8a559adcf2424"
format_stamp: "Formatted at 2026-09-26 03:37:16 on dist-test-slave-dt7p"
server_key: "15eb480e6ce1d7b622e975af0f39e9c1"
server_key_iv: "decec8a85972c5bed23d50387e659103"
server_key_version: "encryptionkey@0"
I20260926 03:37:16.498039 22931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:37:16.506994 22849 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42499 } attempt: 4
I20260926 03:37:16.508396 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:16.507781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42499 (local address 127.0.0.1:49844)
0926 03:37:16.508009 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:16.508132 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 03:37:16.509675 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:16.509323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42499 (local address 127.0.0.1:49854)
0926 03:37:16.509471 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:16.509542 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 03:37:16.510008 22849 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:42499: Network error: Client connection negotiation failed: client connection to 127.0.97.60:42499: connect: Connection refused (error 111)
I20260926 03:37:16.512151 22931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:37:16.516841 22931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:37:16.517151 22931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:37:16.538944 22931 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:42499
I20260926 03:37:16.538969 22997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:42499 every 8 connection(s)
I20260926 03:37:16.540680 22931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-2/data/info.pb
I20260926 03:37:16.545354 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 22931
I20260926 03:37:16.545444 22998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:16.545608 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790393463219689-388-0/minicluster-data/master-2/wal/instance
I20260926 03:37:16.546135 388 external_mini_cluster.cc:1428] Setting key 3fc1622446cbfd9c08c35f852513c3eb
I20260926 03:37:16.555392 22998 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:37:16 dist-test-slave-dt7p krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 26 03:37:16 dist-test-slave-dt7p krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:37:16.571007 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:16.548210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49864 (local address 127.0.97.60:42499)
0926 03:37:16.549190 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:16.549207 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:16.549257 (+ 50us) server_negotiation.cc:407] Connection header received
0926 03:37:16.551034 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:16.551059 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:16.551503 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:16.551973 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:16.553587 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:16.554959 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:16.556426 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:16.557209 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:16.557414 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:16.562562 (+ 5148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:16.562571 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:16.562583 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:16.562635 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:16.566216 (+ 3581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:16.567131 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:16.567138 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:16.567145 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:16.567483 (+ 338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:16.567943 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:16.567947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:16.567949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:16.568521 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:16.568871 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:16.569361 (+ 490us) server_negotiation.cc:299] Negotiation successful
0926 03:37:16.569812 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":749,"thread_start_us":475,"threads_started":1}
I20260926 03:37:16.575047 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:16.558245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35418 (local address 127.0.97.62:39963)
0926 03:37:16.559006 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:16.559014 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:16.559497 (+ 483us) server_negotiation.cc:407] Connection header received
0926 03:37:16.559740 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:16.559746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:16.559837 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:16.559970 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:16.561499 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:16.562371 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:16.563946 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:16.564371 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:16.564516 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:16.568437 (+ 3921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:16.568443 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:16.568446 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:16.568485 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:16.571692 (+ 3207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:16.572437 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:16.572441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:16.572444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:16.572635 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:16.573043 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:16.573046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:16.573049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:16.573323 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:16.573442 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:16.574233 (+ 791us) server_negotiation.cc:299] Negotiation successful
0926 03:37:16.574485 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":393,"threads_started":1}
I20260926 03:37:16.575731 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:16.558280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.0.1:35418)
0926 03:37:16.559347 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:16.559385 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:16.559579 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:16.560037 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:16.560044 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:16.560893 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:16.561299 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:16.561312 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:16.562551 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:16.562560 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:16.563755 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:16.563773 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:16.564625 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:16.564633 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:16.564825 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:16.565493 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:16.565517 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:16.568204 (+ 2687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:16.571880 (+ 3676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:16.571891 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:16.571899 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:16.572301 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:16.572748 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:16.572753 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:16.572756 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:16.572934 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:16.573448 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:16.573455 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:16.573661 (+ 206us) client_negotiation.cc:769] Sending connection context
0926 03:37:16.573897 (+ 236us) client_negotiation.cc:240] Negotiation successful
0926 03:37:16.575415 (+ 1518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":848,"thread_start_us":675,"threads_started":1}
I20260926 03:37:16.577407 22998 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44135 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:37:16 dist-test-slave-dt7p krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 26 03:37:16 dist-test-slave-dt7p krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 03:37:16.591970 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:16.574694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35422 (local address 127.0.97.62:39963)
0926 03:37:16.575356 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:16.575362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:16.575379 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:37:16.575429 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:16.575434 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:16.575521 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:16.575640 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:16.577187 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:16.577980 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:16.579759 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:16.580156 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:16.580246 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:16.585663 (+ 5417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:16.585671 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:16.585675 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:16.585721 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:16.588751 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:16.589660 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:16.589664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:16.589667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:16.589835 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:16.590255 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:16.590260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:16.590263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:16.590674 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:16.590930 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:16.591349 (+ 419us) server_negotiation.cc:299] Negotiation successful
0926 03:37:16.591580 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":376,"threads_started":1}
I20260926 03:37:16.593024 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:16.578250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33186 (local address 127.0.97.61:44135)
0926 03:37:16.578457 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:16.578464 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:16.578489 (+ 25us) server_negotiation.cc:407] Connection header received
0926 03:37:16.578653 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:16.578659 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:16.578748 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:16.578860 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:16.579540 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:16.580444 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:16.581740 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:16.582480 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:16.582636 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:16.586409 (+ 3773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:16.586418 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:16.586425 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:16.586476 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:16.589834 (+ 3358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:16.590700 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:16.590704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:16.590706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:16.590903 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:16.591356 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:16.591361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:16.591364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:16.591758 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:16.591903 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:16.592448 (+ 545us) server_negotiation.cc:299] Negotiation successful
0926 03:37:16.592733 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 03:37:16.593020 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:16.578253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44135 (local address 127.0.0.1:33186)
0926 03:37:16.578413 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:16.578432 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:16.578526 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:16.578881 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:16.578884 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:16.579204 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:16.579384 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:16.579392 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:16.580652 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:16.580656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:16.581586 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:16.581597 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:16.582662 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:16.582667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:16.582833 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:16.583426 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:16.583469 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:16.586117 (+ 2648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:16.589977 (+ 3860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:16.589986 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:16.589990 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:16.590564 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:16.591048 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:16.591052 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:16.591054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:16.591232 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:16.591956 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:16.591959 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:16.592169 (+ 210us) client_negotiation.cc:769] Sending connection context
0926 03:37:16.592311 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 03:37:16.592621 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 03:37:16.594236 22998 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42499 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:37:16 dist-test-slave-dt7p krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 26 03:37:16 dist-test-slave-dt7p krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 03:37:16.608733 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:16.595054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42499 (local address 127.0.0.1:49880)
0926 03:37:16.595267 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:16.595281 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:16.595406 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:16.595718 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:16.595722 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:16.596128 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:16.596271 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:16.596280 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:16.597308 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:16.597316 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:16.598191 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:16.598203 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:16.598902 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:16.598907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:16.599172 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:16.599929 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:16.599946 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:16.602548 (+ 2602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:16.605977 (+ 3429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:16.605984 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:16.605988 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:16.606345 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:16.606746 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:16.606751 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:16.606754 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:16.606931 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:16.607514 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:16.607519 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:16.607662 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 03:37:16.607887 (+ 225us) client_negotiation.cc:240] Negotiation successful
0926 03:37:16.608215 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":16}
I20260926 03:37:16.608731 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:16.595054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49880 (local address 127.0.97.60:42499)
0926 03:37:16.595231 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:16.595237 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:16.595354 (+ 117us) server_negotiation.cc:407] Connection header received
0926 03:37:16.595527 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:16.595533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:16.595599 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:16.595696 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:16.596403 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:16.597176 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:16.598324 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:16.598742 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:16.598886 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:16.602725 (+ 3839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:16.602730 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:16.602734 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:16.602762 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:16.605825 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:16.606471 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:16.606477 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:16.606480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:16.606634 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:16.607069 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:16.607074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:16.607077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:16.607389 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:16.607542 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:16.608038 (+ 496us) server_negotiation.cc:299] Negotiation successful
0926 03:37:16.608396 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":127,"server-negotiator.queue_time_us":58}
I20260926 03:37:16.609966 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:16.594433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33198 (local address 127.0.97.61:44135)
0926 03:37:16.594632 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:16.594637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:16.594655 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:37:16.594819 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:16.594825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:16.594895 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:16.595007 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:16.596269 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:16.597131 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:16.598351 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:16.598797 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:16.598938 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:16.604022 (+ 5084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:16.604027 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:16.604031 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:16.604065 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:16.606981 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:16.607701 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:16.607708 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:16.607712 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:16.607878 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:16.608343 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:16.608351 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:16.608355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:16.608745 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:16.609260 (+ 515us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:16.609405 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 03:37:16.609652 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:37:16.615950 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:37:16.616006 22998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8709711448fe472c83f8a559adcf2424: Bootstrap starting.
I20260926 03:37:16.619982 22998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8709711448fe472c83f8a559adcf2424: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:16.621285 22998 log.cc:824] T 00000000000000000000000000000000 P 8709711448fe472c83f8a559adcf2424: Log is configured to *not* fsync() on all Append() calls
I20260926 03:37:16.618983 22824 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "786111c01d7f4c91abdcff6155c42ec5"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35422
I20260926 03:37:16.624923 22998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8709711448fe472c83f8a559adcf2424: No bootstrap required, opened a new log
I20260926 03:37:16.633363 22998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8709711448fe472c83f8a559adcf2424 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786111c01d7f4c91abdcff6155c42ec5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "ae05067a21c644028932cc74e390c913" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44135 } } peers { permanent_uuid: "8709711448fe472c83f8a559adcf2424" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42499 } }
I20260926 03:37:16.633749 22998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8709711448fe472c83f8a559adcf2424 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8709711448fe472c83f8a559adcf2424, State: Initialized, Role: FOLLOWER
I20260926 03:37:16.634305 22998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8709711448fe472c83f8a559adcf2424 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786111c01d7f4c91abdcff6155c42ec5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "ae05067a21c644028932cc74e390c913" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44135 } } peers { permanent_uuid: "8709711448fe472c83f8a559adcf2424" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42499 } }
I20260926 03:37:16.635416 23006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8709711448fe472c83f8a559adcf2424 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786111c01d7f4c91abdcff6155c42ec5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "ae05067a21c644028932cc74e390c913" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44135 } } peers { permanent_uuid: "8709711448fe472c83f8a559adcf2424" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42499 } } }
I20260926 03:37:16.635849 23006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8709711448fe472c83f8a559adcf2424 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:37:16.636648 22998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8709711448fe472c83f8a559adcf2424 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 03:37:16.647629 23017 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8709711448fe472c83f8a559adcf2424: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:37:16.647830 23017 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:37:16.645195 22931 master_runner.cc:186] Error getting master registration for 127.0.97.62:39963: OK, instance_id { permanent_uuid: "786111c01d7f4c91abdcff6155c42ec5" instance_seqno: 1790393835352954 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 03:37:16.653297 22931 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 03:37:16.678838 22923 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42499 } attempt: 5
Sep 26 03:37:16 dist-test-slave-dt7p krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 03:37:16.698590 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:16.679884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49892 (local address 127.0.97.60:42499)
0926 03:37:16.680174 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:16.680180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:16.680205 (+ 25us) server_negotiation.cc:407] Connection header received
0926 03:37:16.680513 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:16.680522 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:16.680694 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:16.680816 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:16.682087 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:16.683486 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:16.685563 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:16.686472 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:16.686620 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:16.691722 (+ 5102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:16.691728 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:16.691733 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:16.691773 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:16.695287 (+ 3514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:16.696122 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:16.696128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:16.696133 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:16.696302 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:16.696836 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:16.696843 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:16.696847 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:16.697222 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:16.697429 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:16.697855 (+ 426us) server_negotiation.cc:299] Negotiation successful
0926 03:37:16.698122 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 03:37:16.698570 22926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:16.679813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42499 (local address 127.0.0.1:49892)
0926 03:37:16.680087 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:16.680115 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:16.680276 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:16.680901 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:16.680907 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:16.681534 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:16.681805 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:16.681904 (+ 99us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:16.683687 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:16.683694 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:16.685381 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:16.685400 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:16.686661 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:16.686668 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:16.686894 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:16.687792 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:16.687822 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:16.691470 (+ 3648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:16.695478 (+ 4008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:16.695485 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:16.695490 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:16.695968 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:16.696439 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:16.696447 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:16.696452 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:16.696712 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:16.697355 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:16.697362 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:16.697562 (+ 200us) client_negotiation.cc:769] Sending connection context
0926 03:37:16.697738 (+ 176us) client_negotiation.cc:240] Negotiation successful
0926 03:37:16.698111 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 03:37:16.705868 22923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae05067a21c644028932cc74e390c913: Bootstrap starting.
I20260926 03:37:16.710950 22923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae05067a21c644028932cc74e390c913: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:16.712467 22923 log.cc:824] T 00000000000000000000000000000000 P ae05067a21c644028932cc74e390c913: Log is configured to *not* fsync() on all Append() calls
I20260926 03:37:16.716322 22923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae05067a21c644028932cc74e390c913: No bootstrap required, opened a new log
I20260926 03:37:16.728374 22923 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae05067a21c644028932cc74e390c913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786111c01d7f4c91abdcff6155c42ec5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "ae05067a21c644028932cc74e390c913" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44135 } } peers { permanent_uuid: "8709711448fe472c83f8a559adcf2424" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42499 } }
I20260926 03:37:16.728886 22923 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae05067a21c644028932cc74e390c913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae05067a21c644028932cc74e390c913, State: Initialized, Role: FOLLOWER
I20260926 03:37:16.729549 22923 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae05067a21c644028932cc74e390c913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786111c01d7f4c91abdcff6155c42ec5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "ae05067a21c644028932cc74e390c913" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44135 } } peers { permanent_uuid: "8709711448fe472c83f8a559adcf2424" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42499 } }
I20260926 03:37:16.730916 23020 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae05067a21c644028932cc74e390c913 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786111c01d7f4c91abdcff6155c42ec5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "ae05067a21c644028932cc74e390c913" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44135 } } peers { permanent_uuid: "8709711448fe472c83f8a559adcf2424" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42499 } } }
I20260926 03:37:16.731415 23020 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae05067a21c644028932cc74e390c913 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:37:16.732367 22923 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae05067a21c644028932cc74e390c913 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 03:37:16.747795 23031 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ae05067a21c644028932cc74e390c913: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:37:16.748054 23031 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:37:16.744212 22856 master_runner.cc:186] Error getting master registration for 127.0.97.62:39963: OK, instance_id { permanent_uuid: "786111c01d7f4c91abdcff6155c42ec5" instance_seqno: 1790393835352954 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 03:37:16.757378 22856 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 03:37:16.769793 22952 tablet.cc:2369] T 00000000000000000000000000000000 P 8709711448fe472c83f8a559adcf2424: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:37:16.787304 22849 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42499 } attempt: 5
Sep 26 03:37:16 dist-test-slave-dt7p krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 03:37:16.806640 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:16.789001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49904 (local address 127.0.97.60:42499)
0926 03:37:16.789234 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:16.789243 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:16.789263 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:37:16.789334 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:16.789339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:16.789416 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:16.789563 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:16.790924 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:16.791805 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:16.794141 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:16.794935 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:16.795083 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:16.799645 (+ 4562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:16.799651 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:16.799655 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:16.799695 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:16.803589 (+ 3894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:16.804305 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:16.804312 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:16.804316 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:16.804496 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:16.804952 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:16.804958 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:16.804962 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:16.805373 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:16.805528 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:16.805916 (+ 388us) server_negotiation.cc:299] Negotiation successful
0926 03:37:16.806160 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 03:37:16.806640 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:16.788346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42499 (local address 127.0.0.1:49904)
0926 03:37:16.788616 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:16.788645 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:16.788829 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:16.789638 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:16.789644 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:16.790392 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:16.790721 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:16.790739 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:16.792002 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:16.792012 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:16.793892 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:16.793917 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:16.795130 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:16.795138 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:16.795355 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:16.796120 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:16.796147 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:16.799415 (+ 3268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:16.803736 (+ 4321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:16.803743 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:16.803747 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:16.804156 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:16.804619 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:16.804624 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:16.804628 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:16.804800 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:16.805496 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:16.805503 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:16.805660 (+ 157us) client_negotiation.cc:769] Sending connection context
0926 03:37:16.805811 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 03:37:16.806139 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 03:37:16.812647 22849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 786111c01d7f4c91abdcff6155c42ec5: Bootstrap starting.
I20260926 03:37:16.817538 22849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 786111c01d7f4c91abdcff6155c42ec5: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:16.819013 22849 log.cc:824] T 00000000000000000000000000000000 P 786111c01d7f4c91abdcff6155c42ec5: Log is configured to *not* fsync() on all Append() calls
I20260926 03:37:16.823133 22849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 786111c01d7f4c91abdcff6155c42ec5: No bootstrap required, opened a new log
I20260926 03:37:16.831445 22849 raft_consensus.cc:348] T 00000000000000000000000000000000 P 786111c01d7f4c91abdcff6155c42ec5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786111c01d7f4c91abdcff6155c42ec5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "ae05067a21c644028932cc74e390c913" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44135 } } peers { permanent_uuid: "8709711448fe472c83f8a559adcf2424" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42499 } }
I20260926 03:37:16.831898 22849 raft_consensus.cc:729] T 00000000000000000000000000000000 P 786111c01d7f4c91abdcff6155c42ec5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 786111c01d7f4c91abdcff6155c42ec5, State: Initialized, Role: FOLLOWER
I20260926 03:37:16.832531 22849 consensus_queue.cc:260] T 00000000000000000000000000000000 P 786111c01d7f4c91abdcff6155c42ec5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786111c01d7f4c91abdcff6155c42ec5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "ae05067a21c644028932cc74e390c913" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44135 } } peers { permanent_uuid: "8709711448fe472c83f8a559adcf2424" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42499 } }
I20260926 03:37:16.834103 23034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 786111c01d7f4c91abdcff6155c42ec5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786111c01d7f4c91abdcff6155c42ec5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "ae05067a21c644028932cc74e390c913" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44135 } } peers { permanent_uuid: "8709711448fe472c83f8a559adcf2424" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42499 } } }
I20260926 03:37:16.834826 23034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 786111c01d7f4c91abdcff6155c42ec5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:37:16.835723 22849 sys_catalog.cc:564] T 00000000000000000000000000000000 P 786111c01d7f4c91abdcff6155c42ec5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:37:16.835726 22824 raft_consensus.cc:482] T 00000000000000000000000000000000 P 786111c01d7f4c91abdcff6155c42ec5 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 03:37:16.836016 22824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 786111c01d7f4c91abdcff6155c42ec5 [term 0 FOLLOWER]: Advancing to term 1
W20260926 03:37:16.840512 22803 tablet.cc:2369] T 00000000000000000000000000000000 P 786111c01d7f4c91abdcff6155c42ec5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:37:16.842087 22824 raft_consensus.cc:504] T 00000000000000000000000000000000 P 786111c01d7f4c91abdcff6155c42ec5 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786111c01d7f4c91abdcff6155c42ec5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "ae05067a21c644028932cc74e390c913" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44135 } } peers { permanent_uuid: "8709711448fe472c83f8a559adcf2424" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42499 } }
I20260926 03:37:16.845613 22898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "786111c01d7f4c91abdcff6155c42ec5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ae05067a21c644028932cc74e390c913"
I20260926 03:37:16.845886 22973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "786111c01d7f4c91abdcff6155c42ec5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8709711448fe472c83f8a559adcf2424"
I20260926 03:37:16.846280 22898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae05067a21c644028932cc74e390c913 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:16.846469 22973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8709711448fe472c83f8a559adcf2424 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:16.850306 22898 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ae05067a21c644028932cc74e390c913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 786111c01d7f4c91abdcff6155c42ec5 in term 1.
I20260926 03:37:16.850508 22973 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8709711448fe472c83f8a559adcf2424 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 786111c01d7f4c91abdcff6155c42ec5 in term 1.
I20260926 03:37:16.851330 22800 leader_election.cc:302] T 00000000000000000000000000000000 P 786111c01d7f4c91abdcff6155c42ec5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 786111c01d7f4c91abdcff6155c42ec5, 8709711448fe472c83f8a559adcf2424; no voters:
I20260926 03:37:16.852274 22824 leader_election.cc:288] T 00000000000000000000000000000000 P 786111c01d7f4c91abdcff6155c42ec5 [CANDIDATE]: Term 1 election: Requested vote from peers ae05067a21c644028932cc74e390c913 (127.0.97.61:44135), 8709711448fe472c83f8a559adcf2424 (127.0.97.60:42499)
I20260926 03:37:16.853792 23034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 786111c01d7f4c91abdcff6155c42ec5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:37:16.858183 23034 raft_consensus.cc:686] T 00000000000000000000000000000000 P 786111c01d7f4c91abdcff6155c42ec5 [term 1 LEADER]: Becoming Leader. State: Replica: 786111c01d7f4c91abdcff6155c42ec5, State: Running, Role: LEADER
I20260926 03:37:16.859159 23034 consensus_queue.cc:237] T 00000000000000000000000000000000 P 786111c01d7f4c91abdcff6155c42ec5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786111c01d7f4c91abdcff6155c42ec5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "ae05067a21c644028932cc74e390c913" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44135 } } peers { permanent_uuid: "8709711448fe472c83f8a559adcf2424" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42499 } }
I20260926 03:37:16.869490 23036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 786111c01d7f4c91abdcff6155c42ec5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 786111c01d7f4c91abdcff6155c42ec5. Latest consensus state: current_term: 1 leader_uuid: "786111c01d7f4c91abdcff6155c42ec5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786111c01d7f4c91abdcff6155c42ec5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "ae05067a21c644028932cc74e390c913" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44135 } } peers { permanent_uuid: "8709711448fe472c83f8a559adcf2424" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42499 } } }
I20260926 03:37:16.870630 23036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 786111c01d7f4c91abdcff6155c42ec5 [sys.catalog]: This master's current role is: LEADER
I20260926 03:37:16.879781 23041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:37:16.897665 23041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:37:16.926019 22973 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8709711448fe472c83f8a559adcf2424 [term 1 FOLLOWER]: Refusing update from remote peer 786111c01d7f4c91abdcff6155c42ec5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:37:16.927639 23036 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 786111c01d7f4c91abdcff6155c42ec5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8709711448fe472c83f8a559adcf2424" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 03:37:16.928869 22877 tablet.cc:2369] T 00000000000000000000000000000000 P ae05067a21c644028932cc74e390c913: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:37:16.946158 22898 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ae05067a21c644028932cc74e390c913 [term 1 FOLLOWER]: Refusing update from remote peer 786111c01d7f4c91abdcff6155c42ec5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:37:16.947460 23036 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 786111c01d7f4c91abdcff6155c42ec5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae05067a21c644028932cc74e390c913" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:37:16.974025 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:16.900412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49910 (local address 127.0.97.60:42499)
0926 03:37:16.900686 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:16.900693 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:16.914959 (+ 14266us) server_negotiation.cc:407] Connection header received
0926 03:37:16.915972 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:16.915980 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:16.916081 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:16.916177 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:16.924493 (+ 8316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:16.925668 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:16.936533 (+ 10865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:16.937263 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:16.937385 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:16.943686 (+ 6301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:16.943695 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:16.943700 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:16.943753 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:16.950683 (+ 6930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:16.970913 (+ 20230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:16.970921 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:16.970925 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:16.971143 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:16.971683 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:16.971689 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:16.971693 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:16.972183 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:16.972513 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:16.973166 (+ 653us) server_negotiation.cc:299] Negotiation successful
0926 03:37:16.973567 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 03:37:16.978370 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:16.894828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35424 (local address 127.0.97.62:39963)
0926 03:37:16.895098 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:16.895107 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:16.914427 (+ 19320us) server_negotiation.cc:407] Connection header received
0926 03:37:16.915567 (+ 1140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:16.915573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:16.915668 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:16.915750 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:16.921235 (+ 5485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:16.922468 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:16.934366 (+ 11898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:16.935111 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:16.935222 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:16.969855 (+ 34633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:16.969864 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:16.969871 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:16.969955 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:16.974302 (+ 4347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:16.975579 (+ 1277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:16.975585 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:16.975588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:16.975754 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:16.976239 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:16.976244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:16.976247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:16.976694 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:16.976983 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:16.977612 (+ 629us) server_negotiation.cc:299] Negotiation successful
0926 03:37:16.977886 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 03:37:16.991961 23020 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae05067a21c644028932cc74e390c913 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 786111c01d7f4c91abdcff6155c42ec5. Latest consensus state: current_term: 1 leader_uuid: "786111c01d7f4c91abdcff6155c42ec5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786111c01d7f4c91abdcff6155c42ec5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "ae05067a21c644028932cc74e390c913" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44135 } } peers { permanent_uuid: "8709711448fe472c83f8a559adcf2424" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42499 } } }
I20260926 03:37:16.992379 23020 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae05067a21c644028932cc74e390c913 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:37:16.993345 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:16.895335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33204 (local address 127.0.97.61:44135)
0926 03:37:16.895543 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:16.895549 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:16.909244 (+ 13695us) server_negotiation.cc:407] Connection header received
0926 03:37:16.909823 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:16.909831 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:16.909936 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:16.910030 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:16.917480 (+ 7450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:16.918984 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:16.921596 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:16.922651 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:16.922799 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:16.965537 (+ 42738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:16.965550 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:16.965556 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:16.965629 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:16.970017 (+ 4388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:16.988955 (+ 18938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:16.988964 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:16.988968 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:16.989280 (+ 312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:16.990945 (+ 1665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:16.990952 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:16.990955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:16.991499 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:16.991840 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:16.992516 (+ 676us) server_negotiation.cc:299] Negotiation successful
0926 03:37:16.992879 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 03:37:17.000049 23006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8709711448fe472c83f8a559adcf2424 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 786111c01d7f4c91abdcff6155c42ec5. Latest consensus state: current_term: 1 leader_uuid: "786111c01d7f4c91abdcff6155c42ec5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786111c01d7f4c91abdcff6155c42ec5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "ae05067a21c644028932cc74e390c913" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44135 } } peers { permanent_uuid: "8709711448fe472c83f8a559adcf2424" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42499 } } }
I20260926 03:37:17.000459 23006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8709711448fe472c83f8a559adcf2424 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:37:17.011821 23054 mvcc.cc:205] Tried to move back new op lower bound from 7333453156006367232 to 7333453155784564736. Current Snapshot: MvccSnapshot[applied={T|T < 7333453156006367232}]
I20260926 03:37:17.019080 23058 mvcc.cc:205] Tried to move back new op lower bound from 7333453156006367232 to 7333453155784564736. Current Snapshot: MvccSnapshot[applied={T|T < 7333453156006367232}]
I20260926 03:37:17.021096 23020 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae05067a21c644028932cc74e390c913 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "786111c01d7f4c91abdcff6155c42ec5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786111c01d7f4c91abdcff6155c42ec5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "ae05067a21c644028932cc74e390c913" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44135 } } peers { permanent_uuid: "8709711448fe472c83f8a559adcf2424" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42499 } } }
I20260926 03:37:17.021421 23020 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae05067a21c644028932cc74e390c913 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:37:17.037402 23006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8709711448fe472c83f8a559adcf2424 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "786111c01d7f4c91abdcff6155c42ec5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786111c01d7f4c91abdcff6155c42ec5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "ae05067a21c644028932cc74e390c913" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44135 } } peers { permanent_uuid: "8709711448fe472c83f8a559adcf2424" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42499 } } }
I20260926 03:37:17.037772 23006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8709711448fe472c83f8a559adcf2424 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:37:17.040722 23034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 786111c01d7f4c91abdcff6155c42ec5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "786111c01d7f4c91abdcff6155c42ec5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786111c01d7f4c91abdcff6155c42ec5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "ae05067a21c644028932cc74e390c913" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44135 } } peers { permanent_uuid: "8709711448fe472c83f8a559adcf2424" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42499 } } }
I20260926 03:37:17.041122 23034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 786111c01d7f4c91abdcff6155c42ec5 [sys.catalog]: This master's current role is: LEADER
I20260926 03:37:17.041587 23034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 786111c01d7f4c91abdcff6155c42ec5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "786111c01d7f4c91abdcff6155c42ec5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "786111c01d7f4c91abdcff6155c42ec5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "ae05067a21c644028932cc74e390c913" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44135 } } peers { permanent_uuid: "8709711448fe472c83f8a559adcf2424" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42499 } } }
I20260926 03:37:17.041870 23034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 786111c01d7f4c91abdcff6155c42ec5 [sys.catalog]: This master's current role is: LEADER
I20260926 03:37:17.042205 23060 mvcc.cc:205] Tried to move back new op lower bound from 7333453156006367232 to 7333453155784564736. Current Snapshot: MvccSnapshot[applied={T|T < 7333453156006367232 or (T in {7333453156006367232})}]
I20260926 03:37:17.054311 23041 catalog_manager.cc:1347] Generated new cluster ID: a51d77b1c9964d0ea0ef4f465b4d4363
I20260926 03:37:17.054452 23041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:37:17.096737 23041 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:37:17.099087 23041 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:37:17.133399 23041 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 786111c01d7f4c91abdcff6155c42ec5: Generated new TSK 0
I20260926 03:37:17.134954 23041 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 03:37:17.301313 23002 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:37:17.277812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35426 (local address 127.0.97.62:39963)
0926 03:37:17.278088 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:17.278098 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:17.298561 (+ 20463us) server_negotiation.cc:407] Connection header received
0926 03:37:17.299178 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:17.299187 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:17.299305 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:17.299398 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:17.300965 (+ 1567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35426: BlockingRecv error: recv got EOF from 127.0.0.1:35426 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260926 03:37:17.301657 22928 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:37:17.277523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33220 (local address 127.0.97.61:44135)
0926 03:37:17.277786 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:17.277794 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:17.298560 (+ 20766us) server_negotiation.cc:407] Connection header received
0926 03:37:17.299231 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:17.299237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:17.299332 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:17.299410 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:17.301408 (+ 1998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33220: BlockingRecv error: recv got EOF from 127.0.0.1:33220 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260926 03:37:17.304877 23001 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:37:17.284145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49916 (local address 127.0.97.60:42499)
0926 03:37:17.284408 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:17.284416 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:17.300687 (+ 16271us) server_negotiation.cc:407] Connection header received
0926 03:37:17.303206 (+ 2519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:17.303213 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:17.303328 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:17.303412 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:17.304555 (+ 1143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49916: BlockingRecv error: recv got EOF from 127.0.0.1:49916 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 03:37:17.333997 22928 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:37:17.331148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33234 (local address 127.0.97.61:44135)
0926 03:37:17.331408 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:17.331416 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:17.331976 (+ 560us) server_negotiation.cc:407] Connection header received
0926 03:37:17.332254 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:17.332261 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:17.332462 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:17.332629 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:17.333650 (+ 1021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33234: BlockingRecv error: recv got EOF from 127.0.0.1:33234 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
W20260926 03:37:17.342233 22928 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:37:17.339970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33238 (local address 127.0.97.61:44135)
0926 03:37:17.340217 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:17.340224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:17.340264 (+ 40us) server_negotiation.cc:407] Connection header received
0926 03:37:17.340645 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:17.340651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:17.340789 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:17.340940 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:17.341915 (+ 975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33238: BlockingRecv error: recv got EOF from 127.0.0.1:33238 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
W20260926 03:37:17.361248 22928 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:37:17.353860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33242 (local address 127.0.97.61:44135)
0926 03:37:17.354139 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:17.354147 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:17.357303 (+ 3156us) server_negotiation.cc:407] Connection header received
0926 03:37:17.357551 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:17.357561 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:17.357718 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:17.357853 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:17.360922 (+ 3069us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33242: BlockingRecv error: recv got EOF from 127.0.0.1:33242 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260926 03:37:17.395118 22928 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:37:17.392226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33248 (local address 127.0.97.61:44135)
0926 03:37:17.392587 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:17.392595 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:17.393248 (+ 653us) server_negotiation.cc:407] Connection header received
0926 03:37:17.393499 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:17.393506 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:17.393620 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:17.393738 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:17.394824 (+ 1086us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33248: BlockingRecv error: recv got EOF from 127.0.0.1:33248 (error 108)
Metrics: {"server-negotiator.queue_time_us":168}
W20260926 03:37:17.410131 22928 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:37:17.407729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33254 (local address 127.0.97.61:44135)
0926 03:37:17.408019 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:17.408028 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:17.408422 (+ 394us) server_negotiation.cc:407] Connection header received
0926 03:37:17.408645 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:17.408653 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:17.408780 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:17.408945 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:17.409805 (+ 860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33254: BlockingRecv error: recv got EOF from 127.0.0.1:33254 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260926 03:37:17.433593 22928 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:37:17.430936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33258 (local address 127.0.97.61:44135)
0926 03:37:17.431210 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:17.431220 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:17.431786 (+ 566us) server_negotiation.cc:407] Connection header received
0926 03:37:17.432043 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:17.432048 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:17.432205 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:17.432338 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:17.433307 (+ 969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33258: BlockingRecv error: recv got EOF from 127.0.0.1:33258 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260926 03:37:17.473250 22928 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:37:17.470863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33266 (local address 127.0.97.61:44135)
0926 03:37:17.471133 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:17.471141 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:17.471521 (+ 380us) server_negotiation.cc:407] Connection header received
0926 03:37:17.471782 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:17.471792 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:17.471880 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:17.472000 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:17.472905 (+ 905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33266: BlockingRecv error: recv got EOF from 127.0.0.1:33266 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260926 03:37:17.549750 22928 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:37:17.547304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33282 (local address 127.0.97.61:44135)
0926 03:37:17.547641 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:17.547650 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:17.548010 (+ 360us) server_negotiation.cc:407] Connection header received
0926 03:37:17.548293 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:17.548300 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:17.548393 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:17.548531 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:17.549478 (+ 947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33282: BlockingRecv error: recv got EOF from 127.0.0.1:33282 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 03:37:17.650312 23017 catalog_manager.cc:1259] Loaded cluster ID: a51d77b1c9964d0ea0ef4f465b4d4363
I20260926 03:37:17.650449 23017 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8709711448fe472c83f8a559adcf2424: loading cluster ID for follower catalog manager: success
I20260926 03:37:17.653404 23017 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8709711448fe472c83f8a559adcf2424: acquiring CA information for follower catalog manager: success
I20260926 03:37:17.655479 23017 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8709711448fe472c83f8a559adcf2424: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 03:37:17.685089 22928 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:37:17.682712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33292 (local address 127.0.97.61:44135)
0926 03:37:17.683009 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:17.683017 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:17.683333 (+ 316us) server_negotiation.cc:407] Connection header received
0926 03:37:17.683514 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:17.683519 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:17.683651 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:17.683771 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:17.684744 (+ 973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33292: BlockingRecv error: recv got EOF from 127.0.0.1:33292 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
I20260926 03:37:17.750787 23031 catalog_manager.cc:1259] Loaded cluster ID: a51d77b1c9964d0ea0ef4f465b4d4363
I20260926 03:37:17.750980 23031 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ae05067a21c644028932cc74e390c913: loading cluster ID for follower catalog manager: success
I20260926 03:37:17.754021 23031 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ae05067a21c644028932cc74e390c913: acquiring CA information for follower catalog manager: success
I20260926 03:37:17.756058 23031 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ae05067a21c644028932cc74e390c913: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 03:37:17.953150 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:17.945626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33308 (local address 127.0.97.61:44135)
0926 03:37:17.945921 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:17.945928 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:17.946061 (+ 133us) server_negotiation.cc:407] Connection header received
0926 03:37:17.946272 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:17.946277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:17.946566 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:17.946723 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:37:17.947334 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:17.948516 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:17.950025 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:17.950723 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:17.950847 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:17.951430 (+ 583us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:37:17.952344 (+ 914us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:37:17.952505 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:17.952720 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 03:37:17.952817 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 03:37:17.967341 23129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:17.960261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49918 (local address 127.0.97.60:42499)
0926 03:37:17.961303 (+ 1042us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:17.961308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:17.961325 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:37:17.961439 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:17.961444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:17.961640 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:17.961804 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:37:17.962235 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:17.963223 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:17.964680 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:17.965168 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:17.965289 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:17.965807 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:37:17.966537 (+ 730us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:37:17.966668 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:17.966814 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 03:37:17.966899 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":927,"thread_start_us":803,"threads_started":1}
I20260926 03:37:17.983003 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:17.974869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33318 (local address 127.0.97.61:44135)
0926 03:37:17.975167 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:17.975177 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:17.976728 (+ 1551us) server_negotiation.cc:407] Connection header received
0926 03:37:17.976940 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:17.976949 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:17.977079 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:17.977184 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:37:17.977923 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:17.979117 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:17.980933 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:17.981402 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:17.981546 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:17.982138 (+ 592us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:37:17.982309 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:37:17.982399 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:17.982538 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 03:37:17.982641 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260926 03:37:17.984298 23129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:17.976137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49926 (local address 127.0.97.60:42499)
0926 03:37:17.976369 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:17.976377 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:17.977794 (+ 1417us) server_negotiation.cc:407] Connection header received
0926 03:37:17.978014 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:17.978019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:17.978117 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:17.978209 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:37:17.978878 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:17.979713 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:17.982186 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:17.982860 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:17.983000 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:17.983451 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:37:17.983621 (+ 170us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:37:17.983689 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:17.983928 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 03:37:17.984018 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 03:37:17.986235 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:17.974828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35434 (local address 127.0.97.62:39963)
0926 03:37:17.977298 (+ 2470us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:17.977304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:17.977326 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:37:17.977481 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:17.977489 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:17.977788 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:17.978052 (+ 264us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:37:17.978661 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:17.979675 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:17.983121 (+ 3446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:17.983710 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:17.983872 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:17.984436 (+ 564us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:37:17.985036 (+ 600us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:37:17.985157 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:17.985897 (+ 740us) server_negotiation.cc:299] Negotiation successful
0926 03:37:17.985997 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2327,"thread_start_us":2199,"threads_started":1}
I20260926 03:37:17.992062 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 22782
I20260926 03:37:17.993094 22844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:37:18.116966 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 22782
I20260926 03:37:18.144104 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 22856
I20260926 03:37:18.145761 22918 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:37:18.272054 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 22856
I20260926 03:37:18.296068 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 22931
I20260926 03:37:18.297559 22993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:37:18.420436 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 22931
2026-09-26T03:37:18Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5038 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:37:18 dist-test-slave-dt7p krb5kdc[23143](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:37:18 dist-test-slave-dt7p krb5kdc[23143](info): set up 2 sockets
Sep 26 03:37:18 dist-test-slave-dt7p krb5kdc[23143](info): commencing operation
krb5kdc: starting...
W20260926 03:37:19.569550 388 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.003s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/krb5kdc/test-user.keytab.
Sep 26 03:37:19 dist-test-slave-dt7p krb5kdc[23143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:37:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:37:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:37:19.884739 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35411
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35411,127.0.97.61:45779,127.0.97.60:35693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:37:20.281404 23156 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 03:37:20.281840 23156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:37:20.281970 23156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:37:20.282111 23156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:37:20.291038 23156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:37:20.291189 23156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:37:20.291320 23156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:37:20.291409 23156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:37:20.291522 23156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:37:20.302263 23156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35411,127.0.97.61:45779,127.0.97.60:35693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:37:20.303926 23156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:37:20.305559 23156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:37:20.318892 23164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:20.319034 23161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:20.319041 23162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:20.320159 23156 server_base.cc:1034] running on GCE node
I20260926 03:37:20.321208 23156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:37:20.322906 23156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:37:20.324136 23156 hybrid_clock.cc:648] HybridClock initialized: now 1790393840324073 us; error 31 us; skew 500 ppm
I20260926 03:37:20.324536 23156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:37:20 dist-test-slave-dt7p krb5kdc[23143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:37:20.333006 23156 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:37:20.335008 23156 webserver.cc:460] Webserver started at http://127.0.97.62:40663/ using document root <none> and password file <none>
I20260926 03:37:20.335971 23156 fs_manager.cc:359] Metadata directory not provided
I20260926 03:37:20.336123 23156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:37:20.336527 23156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:37:20.340358 23156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-0/data/instance:
uuid: "c744ec68c68a481eac6ca1223d9b1701"
format_stamp: "Formatted at 2026-09-26 03:37:20 on dist-test-slave-dt7p"
server_key: "ee992eab4d594ec55945772f2c7e1967"
server_key_iv: "fc0a0974778c234ce687c40fe351d645"
server_key_version: "encryptionkey@0"
I20260926 03:37:20.341347 23156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-0/wal/instance:
uuid: "c744ec68c68a481eac6ca1223d9b1701"
format_stamp: "Formatted at 2026-09-26 03:37:20 on dist-test-slave-dt7p"
server_key: "ee992eab4d594ec55945772f2c7e1967"
server_key_iv: "fc0a0974778c234ce687c40fe351d645"
server_key_version: "encryptionkey@0"
I20260926 03:37:20.347543 23156 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 03:37:20.351528 23171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:20.353097 23156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:37:20.353313 23156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-0/wal
uuid: "c744ec68c68a481eac6ca1223d9b1701"
format_stamp: "Formatted at 2026-09-26 03:37:20 on dist-test-slave-dt7p"
server_key: "ee992eab4d594ec55945772f2c7e1967"
server_key_iv: "fc0a0974778c234ce687c40fe351d645"
server_key_version: "encryptionkey@0"
I20260926 03:37:20.353555 23156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:37:20.388224 23156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:37:20.393198 23156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:37:20.393528 23156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:37:20.414510 23156 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35411
I20260926 03:37:20.414532 23222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35411 every 8 connection(s)
I20260926 03:37:20.416134 23156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:37:20.418890 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 23156
I20260926 03:37:20.419108 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-0/wal/instance
I20260926 03:37:20.419744 388 external_mini_cluster.cc:1428] Setting key c4b30481677364ef736f5d050654334d
I20260926 03:37:20.421355 23223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:20.432653 23223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35411 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:37:20 dist-test-slave-dt7p krb5kdc[23143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260926 03:37:20.459663 23227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:20.439144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38558 (local address 127.0.97.62:35411)
0926 03:37:20.439957 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:20.439968 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:20.440037 (+ 69us) server_negotiation.cc:407] Connection header received
0926 03:37:20.441524 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:20.441531 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:20.441825 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:20.442067 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:20.443856 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:20.445278 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:20.446799 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:20.447419 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:20.447638 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:20.451479 (+ 3841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:20.451489 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:20.451504 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:20.451538 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:20.454716 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:20.455448 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:20.455460 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:20.455462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:20.455617 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:20.456026 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:20.456031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:20.456034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:20.456710 (+ 676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:20.456945 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:20.457604 (+ 659us) server_negotiation.cc:299] Negotiation successful
0926 03:37:20.458055 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":379,"threads_started":1}
I20260926 03:37:20.459707 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:20.437494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35411 (local address 127.0.0.1:38558)
0926 03:37:20.438532 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:20.438578 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:20.441061 (+ 2483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:20.442101 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:20.442109 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:20.443110 (+ 1001us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:20.443604 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:20.443625 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:20.445448 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:20.445454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:20.446621 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:20.446638 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:20.447678 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:20.447685 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:20.447891 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:20.448706 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:20.448739 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:20.451247 (+ 2508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:20.454854 (+ 3607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:20.454868 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:20.454884 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:20.455297 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:20.455731 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:20.455736 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:20.455740 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:20.455910 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:20.456848 (+ 938us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:20.456862 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:20.457098 (+ 236us) client_negotiation.cc:769] Sending connection context
0926 03:37:20.457434 (+ 336us) client_negotiation.cc:240] Negotiation successful
0926 03:37:20.458312 (+ 878us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"mutex_wait_us":199,"thread_start_us":435,"threads_started":1}
I20260926 03:37:20.463388 23223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45779 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:37:20.464521 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:20.464111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45779 (local address 127.0.0.1:36840)
0926 03:37:20.464285 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:20.464402 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 03:37:20.464838 23175 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45779: connect: Connection refused (error 111)
I20260926 03:37:20.466530 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:20.466118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45779 (local address 127.0.0.1:36856)
0926 03:37:20.466280 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:20.466371 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 03:37:20.466862 23223 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45779: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45779: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260926 03:37:20.495893 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:45779
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35411,127.0.97.61:45779,127.0.97.60:35693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:37:20.516252 23223 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45779 } attempt: 1
I20260926 03:37:20.517769 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:20.517254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45779 (local address 127.0.0.1:36860)
0926 03:37:20.517479 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:20.517591 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 03:37:20.518973 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:20.518571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45779 (local address 127.0.0.1:36874)
0926 03:37:20.518725 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:20.518809 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 03:37:20.519167 23223 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45779: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45779: connect: Connection refused (error 111)
I20260926 03:37:20.587467 23223 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45779 } attempt: 2
I20260926 03:37:20.588958 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:20.588373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45779 (local address 127.0.0.1:36884)
0926 03:37:20.588660 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:20.588777 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 03:37:20.590011 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:20.589652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45779 (local address 127.0.0.1:36896)
0926 03:37:20.589788 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:20.589880 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 03:37:20.590204 23223 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45779: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45779: connect: Connection refused (error 111)
I20260926 03:37:20.681481 23223 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45779 } attempt: 3
I20260926 03:37:20.682977 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:20.682417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45779 (local address 127.0.0.1:36912)
0926 03:37:20.682661 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:20.682800 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260926 03:37:20.684134 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:20.683744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45779 (local address 127.0.0.1:36920)
0926 03:37:20.683897 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:20.683976 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 03:37:20.684331 23223 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45779: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45779: connect: Connection refused (error 111)
I20260926 03:37:20.827704 23223 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45779 } attempt: 4
I20260926 03:37:20.829357 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:20.828691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45779 (local address 127.0.0.1:36930)
0926 03:37:20.829093 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:20.829200 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216}
I20260926 03:37:20.830456 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:20.830098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45779 (local address 127.0.0.1:36938)
0926 03:37:20.830243 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:20.830334 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 03:37:20.830644 23223 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45779: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45779: connect: Connection refused (error 111)
W20260926 03:37:20.878953 23230 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 03:37:20.879357 23230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:37:20.879491 23230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:37:20.879637 23230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:37:20.887329 23230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:37:20.887456 23230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:37:20.887588 23230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:37:20.887676 23230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:37:20.887794 23230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:37:20.898263 23230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35411,127.0.97.61:45779,127.0.97.60:35693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:45779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:37:20.899852 23230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:37:20.901474 23230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:37:20.914558 23235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:20.914745 23236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:20.914772 23238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:20.916018 23230 server_base.cc:1034] running on GCE node
I20260926 03:37:20.916949 23230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:37:20.918493 23230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:37:20.919739 23230 hybrid_clock.cc:648] HybridClock initialized: now 1790393840919686 us; error 35 us; skew 500 ppm
I20260926 03:37:20.920096 23230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:37:20 dist-test-slave-dt7p krb5kdc[23143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:37:20.923507 23230 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260926 03:37:20.925143 23230 webserver.cc:460] Webserver started at http://127.0.97.61:36437/ using document root <none> and password file <none>
I20260926 03:37:20.925957 23230 fs_manager.cc:359] Metadata directory not provided
I20260926 03:37:20.926074 23230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:37:20.926385 23230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:37:20.929700 23230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-1/data/instance:
uuid: "3905cb4d9bf24e3593154f6c13c86855"
format_stamp: "Formatted at 2026-09-26 03:37:20 on dist-test-slave-dt7p"
server_key: "442c803b795830c4ce150e6868980cce"
server_key_iv: "1d50053307e574f492e6c79de336fb7f"
server_key_version: "encryptionkey@0"
I20260926 03:37:20.930495 23230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-1/wal/instance:
uuid: "3905cb4d9bf24e3593154f6c13c86855"
format_stamp: "Formatted at 2026-09-26 03:37:20 on dist-test-slave-dt7p"
server_key: "442c803b795830c4ce150e6868980cce"
server_key_iv: "1d50053307e574f492e6c79de336fb7f"
server_key_version: "encryptionkey@0"
I20260926 03:37:20.936237 23230 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:37:20.939996 23245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:20.941594 23230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 03:37:20.941798 23230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-1/wal
uuid: "3905cb4d9bf24e3593154f6c13c86855"
format_stamp: "Formatted at 2026-09-26 03:37:20 on dist-test-slave-dt7p"
server_key: "442c803b795830c4ce150e6868980cce"
server_key_iv: "1d50053307e574f492e6c79de336fb7f"
server_key_version: "encryptionkey@0"
I20260926 03:37:20.942063 23230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:37:20.961030 23230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:37:20.965521 23230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:37:20.965828 23230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:37:20.986805 23230 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:45779
I20260926 03:37:20.986830 23296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:45779 every 8 connection(s)
I20260926 03:37:20.988462 23230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-1/data/info.pb
I20260926 03:37:20.993121 23297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:20.997666 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 23230
I20260926 03:37:20.997900 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-1/wal/instance
I20260926 03:37:20.998435 388 external_mini_cluster.cc:1428] Setting key 6e06aa1153721aeee43f244242b226e4
I20260926 03:37:21.000906 23297 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35411 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:37:21 dist-test-slave-dt7p krb5kdc[23143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:37:21.025478 23299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.003999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38574 (local address 127.0.97.62:35411)
0926 03:37:21.004684 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:21.004693 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:21.005829 (+ 1136us) server_negotiation.cc:407] Connection header received
0926 03:37:21.007997 (+ 2168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:21.008007 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:21.008124 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:21.008282 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:21.010638 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.011421 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.013961 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.014410 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.014516 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:21.019079 (+ 4563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:21.019084 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:21.019088 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:21.019140 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:21.022091 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:21.022843 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:21.022853 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:21.022855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:21.022982 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:21.023417 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:21.023428 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:21.023431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:21.023759 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:21.023923 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:21.024958 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0926 03:37:21.025145 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":320,"threads_started":1}
I20260926 03:37:21.027333 23300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.004684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35411 (local address 127.0.0.1:38574)
0926 03:37:21.005618 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:21.005679 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:21.007675 (+ 1996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:21.008602 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:21.008618 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:21.009537 (+ 919us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:21.010413 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:21.010440 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.011594 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.011603 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:21.013727 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:21.013745 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.014657 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.014666 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:21.015009 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:21.015844 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:21.015880 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:21.018821 (+ 2941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:21.022229 (+ 3408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:21.022241 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:21.022259 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:21.022707 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:21.023096 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:21.023102 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:21.023106 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:21.023300 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:21.023925 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:21.023938 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:21.024467 (+ 529us) client_negotiation.cc:769] Sending connection context
0926 03:37:21.024882 (+ 415us) client_negotiation.cc:240] Negotiation successful
0926 03:37:21.025561 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":398,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260926 03:37:21.029424 23297 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45779 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 26 03:37:21 dist-test-slave-dt7p krb5kdc[23143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 03:37:21.046583 23300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.030164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45779 (local address 127.0.0.1:36942)
0926 03:37:21.030351 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:21.030368 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:21.030457 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:21.031451 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:21.031455 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:21.031956 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:21.032161 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:21.032172 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.033664 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.033673 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:21.035231 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:21.035249 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.036135 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.036143 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:21.036375 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:21.037059 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:21.037088 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:21.039796 (+ 2708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:21.043747 (+ 3951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:21.043751 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:21.043754 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:21.044125 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:21.044550 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:21.044555 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:21.044559 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:21.044753 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:21.045431 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:21.045437 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:21.045610 (+ 173us) client_negotiation.cc:769] Sending connection context
0926 03:37:21.045777 (+ 167us) client_negotiation.cc:240] Negotiation successful
0926 03:37:21.046148 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 03:37:21.046631 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.030384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36942 (local address 127.0.97.61:45779)
0926 03:37:21.031021 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:21.031028 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:21.031054 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:37:21.031095 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:21.031101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:21.031298 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:21.031462 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:21.032356 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.033502 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.035389 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.035970 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.036069 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:21.040019 (+ 3950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:21.040032 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:21.040050 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:21.040085 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:21.043600 (+ 3515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:21.044258 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:21.044265 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:21.044268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:21.044430 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:21.044873 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:21.044881 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:21.044886 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:21.045304 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:21.045549 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:21.045948 (+ 399us) server_negotiation.cc:299] Negotiation successful
0926 03:37:21.046255 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":482,"thread_start_us":347,"threads_started":1}
I20260926 03:37:21.049700 23297 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35693 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:37:21.051070 23300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.050563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35693 (local address 127.0.0.1:50286)
0926 03:37:21.050778 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:21.050903 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 03:37:21.051440 23248 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35693: connect: Connection refused (error 111)
I20260926 03:37:21.053320 23300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.052849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35693 (local address 127.0.0.1:50296)
0926 03:37:21.053040 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:21.053156 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 03:37:21.053560 23297 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35693: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35693: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260926 03:37:21.070407 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskssCcLE/build/asan/bin/kudu
/tmp/dist-test-taskssCcLE/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:35693
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35411,127.0.97.61:45779,127.0.97.60:35693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:37:21.102954 23297 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35693 } attempt: 1
I20260926 03:37:21.104439 23300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.103882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35693 (local address 127.0.0.1:50308)
0926 03:37:21.104114 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:21.104230 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 03:37:21.105631 23300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.105247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35693 (local address 127.0.0.1:50312)
0926 03:37:21.105422 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:21.105510 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 03:37:21.105885 23297 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35693: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35693: connect: Connection refused (error 111)
I20260926 03:37:21.129897 23223 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45779 } attempt: 5
Sep 26 03:37:21 dist-test-slave-dt7p krb5kdc[23143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 03:37:21.145895 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.130936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45779 (local address 127.0.0.1:36950)
0926 03:37:21.131195 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:21.131220 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:21.131436 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:21.131839 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:21.131844 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:21.132357 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:21.132637 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:21.132649 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.133738 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.133743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:21.134667 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:21.134679 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.135517 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.135523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:21.135736 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:21.136698 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:21.136749 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:21.139611 (+ 2862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:21.143538 (+ 3927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:21.143543 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:21.143546 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:21.143928 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:21.144315 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:21.144318 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:21.144321 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:21.144448 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:21.145093 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:21.145097 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:21.145219 (+ 122us) client_negotiation.cc:769] Sending connection context
0926 03:37:21.145349 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 03:37:21.145598 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 03:37:21.146054 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.130960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36950 (local address 127.0.97.61:45779)
0926 03:37:21.131220 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:21.131227 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:21.131338 (+ 111us) server_negotiation.cc:407] Connection header received
0926 03:37:21.131598 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:21.131603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:21.131681 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:21.131805 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:21.132789 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.133591 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.134821 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.135328 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.135454 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:21.139830 (+ 4376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:21.139837 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:21.139842 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:21.139890 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:21.143410 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:21.144038 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:21.144044 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:21.144048 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:21.144199 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:21.144545 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:21.144550 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:21.144553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:21.144983 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:21.145143 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:21.145486 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 03:37:21.145725 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260926 03:37:21.147270 23223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35693 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:37:21.148620 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.148176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35693 (local address 127.0.0.1:50320)
0926 03:37:21.148344 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:21.148425 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 03:37:21.149655 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.149353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35693 (local address 127.0.0.1:50336)
0926 03:37:21.149487 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:21.149557 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 03:37:21.149875 23223 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35693: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35693: connect: Connection refused (error 111)
I20260926 03:37:21.174126 23297 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35693 } attempt: 2
I20260926 03:37:21.175607 23300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.175064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35693 (local address 127.0.0.1:50350)
0926 03:37:21.175291 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:21.175413 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 03:37:21.176783 23300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.176377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35693 (local address 127.0.0.1:50352)
0926 03:37:21.176550 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:21.176659 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 03:37:21.177013 23297 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35693: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35693: connect: Connection refused (error 111)
I20260926 03:37:21.201121 23223 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35693 } attempt: 1
I20260926 03:37:21.202759 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.202044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35693 (local address 127.0.0.1:50364)
0926 03:37:21.202340 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:21.202493 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260926 03:37:21.203949 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.203562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35693 (local address 127.0.0.1:50370)
0926 03:37:21.203717 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:21.203809 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 03:37:21.204193 23223 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35693: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35693: connect: Connection refused (error 111)
I20260926 03:37:21.268260 23297 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35693 } attempt: 3
I20260926 03:37:21.269754 23300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.269158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35693 (local address 127.0.0.1:50384)
0926 03:37:21.269435 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:21.269551 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 03:37:21.271024 23300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.270591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35693 (local address 127.0.0.1:50390)
0926 03:37:21.270840 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:21.270913 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260926 03:37:21.271232 23297 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35693: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35693: connect: Connection refused (error 111)
I20260926 03:37:21.272485 23223 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35693 } attempt: 2
I20260926 03:37:21.273810 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.273196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35693 (local address 127.0.0.1:50402)
0926 03:37:21.273437 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:21.273558 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 03:37:21.274892 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.274488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35693 (local address 127.0.0.1:50412)
0926 03:37:21.274655 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:21.274743 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 03:37:21.275121 23223 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35693: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35693: connect: Connection refused (error 111)
I20260926 03:37:21.381387 23223 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35693 } attempt: 3
I20260926 03:37:21.382987 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.382301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35693 (local address 127.0.0.1:50414)
0926 03:37:21.382584 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:21.382735 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260926 03:37:21.384326 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.383783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35693 (local address 127.0.0.1:50422)
0926 03:37:21.384027 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:21.384146 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260926 03:37:21.384598 23223 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35693: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35693: connect: Connection refused (error 111)
I20260926 03:37:21.414479 23297 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35693 } attempt: 4
I20260926 03:37:21.415863 23300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.415295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35693 (local address 127.0.0.1:50430)
0926 03:37:21.415580 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:21.415678 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 03:37:21.417069 23300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.416719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35693 (local address 127.0.0.1:50432)
0926 03:37:21.416875 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:21.416950 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 03:37:21.417348 23297 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35693: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35693: connect: Connection refused (error 111)
W20260926 03:37:21.471354 23305 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 03:37:21.471772 23305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:37:21.471902 23305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:37:21.472048 23305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:37:21.479722 23305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:37:21.479836 23305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:37:21.479931 23305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:37:21.480013 23305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:37:21.480134 23305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:37:21.490950 23305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35411,127.0.97.61:45779,127.0.97.60:35693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:35693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:34:42 UTC on ccycloud.quasar-ockqre.root.comops.site
build id 4673639
ASAN enabled
I20260926 03:37:21.492666 23305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:37:21.494275 23305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:37:21.508342 23311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:21.508289 23313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:37:21.508390 23310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:37:21.514159 23305 server_base.cc:1034] running on GCE node
I20260926 03:37:21.515051 23305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:37:21.516647 23305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:37:21.517920 23305 hybrid_clock.cc:648] HybridClock initialized: now 1790393841517859 us; error 24 us; skew 500 ppm
I20260926 03:37:21.518337 23305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:37:21 dist-test-slave-dt7p krb5kdc[23143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:37:21.521960 23305 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260926 03:37:21.523550 23305 webserver.cc:460] Webserver started at http://127.0.97.60:38225/ using document root <none> and password file <none>
I20260926 03:37:21.524377 23305 fs_manager.cc:359] Metadata directory not provided
I20260926 03:37:21.524508 23305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:37:21.524919 23305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:37:21.528393 23305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-2/data/instance:
uuid: "c48aac0cfd624b8a8d7aa8a348f112fe"
format_stamp: "Formatted at 2026-09-26 03:37:21 on dist-test-slave-dt7p"
server_key: "238b26b67377364efbf8a038c77a5b0c"
server_key_iv: "8baae0a638a9f8d445e36e6eb648cd14"
server_key_version: "encryptionkey@0"
I20260926 03:37:21.529304 23305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-2/wal/instance:
uuid: "c48aac0cfd624b8a8d7aa8a348f112fe"
format_stamp: "Formatted at 2026-09-26 03:37:21 on dist-test-slave-dt7p"
server_key: "238b26b67377364efbf8a038c77a5b0c"
server_key_iv: "8baae0a638a9f8d445e36e6eb648cd14"
server_key_version: "encryptionkey@0"
I20260926 03:37:21.534926 23305 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:37:21.538690 23320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:37:21.540369 23305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:37:21.540661 23305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-2/wal
uuid: "c48aac0cfd624b8a8d7aa8a348f112fe"
format_stamp: "Formatted at 2026-09-26 03:37:21 on dist-test-slave-dt7p"
server_key: "238b26b67377364efbf8a038c77a5b0c"
server_key_iv: "8baae0a638a9f8d445e36e6eb648cd14"
server_key_version: "encryptionkey@0"
I20260926 03:37:21.540930 23305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:37:21.554921 23305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:37:21.560937 23305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:37:21.561232 23305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:37:21.561820 23223 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35693 } attempt: 4
I20260926 03:37:21.564975 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.564327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35693 (local address 127.0.0.1:50436)
0926 03:37:21.564609 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:21.564734 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 03:37:21.566452 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.566033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35693 (local address 127.0.0.1:50444)
0926 03:37:21.566187 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:21.566284 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 03:37:21.567386 23223 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35693: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35693: connect: Connection refused (error 111)
I20260926 03:37:21.585086 23305 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:35693
I20260926 03:37:21.585106 23371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:35693 every 8 connection(s)
I20260926 03:37:21.586804 23305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-2/data/info.pb
I20260926 03:37:21.591373 23372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:37:21.594830 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskssCcLE/build/asan/bin/kudu as pid 23305
I20260926 03:37:21.595067 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskssCcLE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790393463219689-388-0/minicluster-data/master-2/wal/instance
I20260926 03:37:21.595570 388 external_mini_cluster.cc:1428] Setting key 09a10c9c595d1c64d1d28a12ed507126
I20260926 03:37:21.600198 23372 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35411 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:37:21 dist-test-slave-dt7p krb5kdc[23143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 26 03:37:21 dist-test-slave-dt7p krb5kdc[23143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 03:37:21.621555 23376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.603627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38582 (local address 127.0.97.62:35411)
0926 03:37:21.604287 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:21.604293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:21.604889 (+ 596us) server_negotiation.cc:407] Connection header received
0926 03:37:21.605096 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:21.605100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:21.605182 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:21.605279 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:21.606722 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.607857 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.609696 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.610118 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.610260 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:21.613911 (+ 3651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:21.613916 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:21.613920 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:21.613967 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:21.617585 (+ 3618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:21.618564 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:21.618568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:21.618571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:21.618703 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:21.619235 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:21.619239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:21.619243 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:21.619589 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:21.619763 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:21.620846 (+ 1083us) server_negotiation.cc:299] Negotiation successful
0926 03:37:21.621107 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":280,"threads_started":1}
I20260926 03:37:21.622686 23375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.597874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50448 (local address 127.0.97.60:35693)
0926 03:37:21.598792 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:21.598808 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:21.598856 (+ 48us) server_negotiation.cc:407] Connection header received
0926 03:37:21.600794 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:21.600822 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:21.601351 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:21.601829 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:21.603704 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.606010 (+ 2306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.607765 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.608864 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.609057 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:21.614544 (+ 5487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:21.614560 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:21.614586 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:21.614622 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:21.618194 (+ 3572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:21.619094 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:21.619101 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:21.619103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:21.619284 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:21.619689 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:21.619693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:21.619696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:21.620158 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:21.620409 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:21.620901 (+ 492us) server_negotiation.cc:299] Negotiation successful
0926 03:37:21.621271 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":669,"thread_start_us":403,"threads_started":1}
I20260926 03:37:21.622713 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.603665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35411 (local address 127.0.0.1:38582)
0926 03:37:21.604715 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:21.604764 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:21.604962 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:21.605399 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:21.605407 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:21.606286 (+ 879us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:21.606555 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:21.606564 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.608003 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.608008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:21.609539 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:21.609549 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.610376 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.610384 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:21.610543 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:21.611192 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:21.611219 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:21.613686 (+ 2467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:21.617728 (+ 4042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:21.617739 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:21.617750 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:21.618426 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:21.618823 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:21.618829 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:21.618833 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:21.619096 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:21.619751 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:21.619758 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:21.620299 (+ 541us) client_negotiation.cc:769] Sending connection context
0926 03:37:21.620697 (+ 398us) client_negotiation.cc:240] Negotiation successful
0926 03:37:21.621600 (+ 903us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"mutex_wait_us":185,"spinlock_wait_cycles":18304,"thread_start_us":523,"threads_started":1}
I20260926 03:37:21.624715 23372 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45779 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:37:21 dist-test-slave-dt7p krb5kdc[23143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 26 03:37:21 dist-test-slave-dt7p krb5kdc[23143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:37:21.639457 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.625938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36956 (local address 127.0.97.61:45779)
0926 03:37:21.626184 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:21.626191 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:21.626215 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:37:21.626296 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:21.626306 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:21.626411 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:21.626528 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:21.627334 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.628064 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.629365 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.629835 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.629977 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:21.633666 (+ 3689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:21.633671 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:21.633674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:21.633710 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:21.636507 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:21.637447 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:21.637451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:21.637453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:21.637590 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:21.638080 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:21.638084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:21.638086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:21.638379 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:21.638498 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:21.638980 (+ 482us) server_negotiation.cc:299] Negotiation successful
0926 03:37:21.639161 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:37:21.639616 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.625453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45779 (local address 127.0.0.1:36956)
0926 03:37:21.625642 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:21.625663 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:21.625778 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:21.626573 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:21.626577 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:21.626998 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:21.627154 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:21.627162 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.628218 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.628224 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:21.629201 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:21.629212 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.629982 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.629987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:21.630192 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:21.630784 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:21.630809 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:21.633458 (+ 2649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:21.636701 (+ 3243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:21.636705 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:21.636708 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:21.637330 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:21.637719 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:21.637728 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:21.637735 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:21.637948 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:21.638502 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:21.638509 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:21.638712 (+ 203us) client_negotiation.cc:769] Sending connection context
0926 03:37:21.638880 (+ 168us) client_negotiation.cc:240] Negotiation successful
0926 03:37:21.639199 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 03:37:21.640950 23372 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35693 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:37:21.642769 23376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.626652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38592 (local address 127.0.97.62:35411)
0926 03:37:21.626846 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:21.626852 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:21.626870 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:37:21.626992 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:21.626997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:21.627094 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:21.627218 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:21.628518 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.629316 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.630650 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.631137 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.631294 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:21.636033 (+ 4739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:21.636038 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:21.636042 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:21.636076 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:21.639649 (+ 3573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:21.640533 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:21.640538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:21.640541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:21.640718 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:21.641128 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:21.641136 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:21.641140 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:21.641513 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:21.641853 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:21.642202 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 03:37:21.642461 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 03:37:21 dist-test-slave-dt7p krb5kdc[23143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 26 03:37:21 dist-test-slave-dt7p krb5kdc[23143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 03:37:21.656342 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.641917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35693 (local address 127.0.0.1:50460)
0926 03:37:21.642086 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:21.642106 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:21.642221 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:21.642540 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:21.642545 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:21.643224 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:21.643380 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:21.643389 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.644685 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.644691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:21.646038 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:21.646051 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.646858 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.646865 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:21.647130 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:21.647703 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:21.647732 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:21.650057 (+ 2325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:21.653551 (+ 3494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:21.653559 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:21.653563 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:21.653959 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:21.654349 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:21.654355 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:21.654363 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:21.654506 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:21.655230 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:21.655234 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:21.655362 (+ 128us) client_negotiation.cc:769] Sending connection context
0926 03:37:21.655493 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 03:37:21.655881 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 03:37:21.656454 23375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.641787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50460 (local address 127.0.97.60:35693)
0926 03:37:21.641968 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:21.641973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:21.642171 (+ 198us) server_negotiation.cc:407] Connection header received
0926 03:37:21.642347 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:21.642352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:21.642422 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:21.642526 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:21.643523 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.644531 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.646188 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.646628 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.646842 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:21.650275 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:21.650280 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:21.650284 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:21.650346 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:21.653414 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:21.654083 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:21.654091 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:21.654095 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:21.654236 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:21.654646 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:21.654650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:21.654652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:21.655082 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:21.655249 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:21.655641 (+ 392us) server_negotiation.cc:299] Negotiation successful
0926 03:37:21.656047 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":147,"server-negotiator.queue_time_us":72}
I20260926 03:37:21.660789 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.644499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36962 (local address 127.0.97.61:45779)
0926 03:37:21.644701 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:21.644727 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:21.644744 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:37:21.644901 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:21.644923 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:21.645001 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:21.645103 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:21.646439 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.647637 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.649250 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.649747 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.649883 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:21.654025 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:21.654030 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:21.654033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:21.654066 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:21.657527 (+ 3461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:21.658562 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:21.658566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:21.658569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:21.658735 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:21.659186 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:21.659191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:21.659194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:21.659602 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:21.659915 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:21.660166 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 03:37:21.660409 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 03:37:21.664316 23372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c48aac0cfd624b8a8d7aa8a348f112fe: Bootstrap starting.
I20260926 03:37:21.666039 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:37:21.668991 23372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c48aac0cfd624b8a8d7aa8a348f112fe: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:21.670228 23372 log.cc:824] T 00000000000000000000000000000000 P c48aac0cfd624b8a8d7aa8a348f112fe: Log is configured to *not* fsync() on all Append() calls
I20260926 03:37:21.669574 23198 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c744ec68c68a481eac6ca1223d9b1701"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38592
I20260926 03:37:21.673674 23372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c48aac0cfd624b8a8d7aa8a348f112fe: No bootstrap required, opened a new log
I20260926 03:37:21.681399 23372 raft_consensus.cc:348] T 00000000000000000000000000000000 P c48aac0cfd624b8a8d7aa8a348f112fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744ec68c68a481eac6ca1223d9b1701" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35411 } } peers { permanent_uuid: "3905cb4d9bf24e3593154f6c13c86855" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45779 } } peers { permanent_uuid: "c48aac0cfd624b8a8d7aa8a348f112fe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35693 } }
I20260926 03:37:21.681865 23372 raft_consensus.cc:729] T 00000000000000000000000000000000 P c48aac0cfd624b8a8d7aa8a348f112fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c48aac0cfd624b8a8d7aa8a348f112fe, State: Initialized, Role: FOLLOWER
I20260926 03:37:21.682380 23372 consensus_queue.cc:260] T 00000000000000000000000000000000 P c48aac0cfd624b8a8d7aa8a348f112fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744ec68c68a481eac6ca1223d9b1701" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35411 } } peers { permanent_uuid: "3905cb4d9bf24e3593154f6c13c86855" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45779 } } peers { permanent_uuid: "c48aac0cfd624b8a8d7aa8a348f112fe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35693 } }
I20260926 03:37:21.683475 23379 sys_catalog.cc:455] T 00000000000000000000000000000000 P c48aac0cfd624b8a8d7aa8a348f112fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744ec68c68a481eac6ca1223d9b1701" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35411 } } peers { permanent_uuid: "3905cb4d9bf24e3593154f6c13c86855" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45779 } } peers { permanent_uuid: "c48aac0cfd624b8a8d7aa8a348f112fe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35693 } } }
I20260926 03:37:21.683929 23379 sys_catalog.cc:458] T 00000000000000000000000000000000 P c48aac0cfd624b8a8d7aa8a348f112fe [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:37:21.684751 23372 sys_catalog.cc:564] T 00000000000000000000000000000000 P c48aac0cfd624b8a8d7aa8a348f112fe [sys.catalog]: configured and running, proceeding with master startup.
W20260926 03:37:21.696058 23390 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c48aac0cfd624b8a8d7aa8a348f112fe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:37:21.696246 23390 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:37:21.693241 23305 master_runner.cc:186] Error getting master registration for 127.0.97.62:35411: OK, instance_id { permanent_uuid: "c744ec68c68a481eac6ca1223d9b1701" instance_seqno: 1790393840408529 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 03:37:21.706591 23305 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 03:37:21.716640 23297 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35693 } attempt: 5
Sep 26 03:37:21 dist-test-slave-dt7p krb5kdc[23143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 03:37:21.734176 23300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.717491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35693 (local address 127.0.0.1:50468)
0926 03:37:21.717763 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:21.717789 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:21.717950 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:21.718495 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:21.718501 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:21.719117 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:21.719379 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:21.719394 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.720478 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.720485 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:21.722029 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:21.722051 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.723178 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.723184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:21.723407 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:21.724216 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:21.724246 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:21.727505 (+ 3259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:21.731506 (+ 4001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:21.731512 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:21.731516 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:21.731921 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:21.732317 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:21.732322 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:21.732326 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:21.732478 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:21.733111 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:21.733117 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:21.733291 (+ 174us) client_negotiation.cc:769] Sending connection context
0926 03:37:21.733447 (+ 156us) client_negotiation.cc:240] Negotiation successful
0926 03:37:21.733773 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 03:37:21.734193 23375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.717625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50468 (local address 127.0.97.60:35693)
0926 03:37:21.717829 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:21.717834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:21.717856 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:37:21.718148 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:21.718154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:21.718310 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:21.718433 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:21.719507 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.720324 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.722207 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.723011 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.723135 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:21.727705 (+ 4570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:21.727710 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:21.727714 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:21.727747 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:21.731369 (+ 3622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:21.732037 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:21.732045 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:21.732048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:21.732200 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:21.732613 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:21.732618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:21.732621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:21.732989 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:21.733181 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:21.733538 (+ 357us) server_negotiation.cc:299] Negotiation successful
0926 03:37:21.733769 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 03:37:21.739409 23297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3905cb4d9bf24e3593154f6c13c86855: Bootstrap starting.
I20260926 03:37:21.743199 23297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3905cb4d9bf24e3593154f6c13c86855: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:21.744519 23297 log.cc:824] T 00000000000000000000000000000000 P 3905cb4d9bf24e3593154f6c13c86855: Log is configured to *not* fsync() on all Append() calls
I20260926 03:37:21.747786 23297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3905cb4d9bf24e3593154f6c13c86855: No bootstrap required, opened a new log
I20260926 03:37:21.759092 23297 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3905cb4d9bf24e3593154f6c13c86855 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744ec68c68a481eac6ca1223d9b1701" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35411 } } peers { permanent_uuid: "3905cb4d9bf24e3593154f6c13c86855" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45779 } } peers { permanent_uuid: "c48aac0cfd624b8a8d7aa8a348f112fe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35693 } }
I20260926 03:37:21.759528 23297 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3905cb4d9bf24e3593154f6c13c86855 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3905cb4d9bf24e3593154f6c13c86855, State: Initialized, Role: FOLLOWER
I20260926 03:37:21.760114 23297 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3905cb4d9bf24e3593154f6c13c86855 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744ec68c68a481eac6ca1223d9b1701" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35411 } } peers { permanent_uuid: "3905cb4d9bf24e3593154f6c13c86855" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45779 } } peers { permanent_uuid: "c48aac0cfd624b8a8d7aa8a348f112fe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35693 } }
I20260926 03:37:21.761243 23393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3905cb4d9bf24e3593154f6c13c86855 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744ec68c68a481eac6ca1223d9b1701" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35411 } } peers { permanent_uuid: "3905cb4d9bf24e3593154f6c13c86855" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45779 } } peers { permanent_uuid: "c48aac0cfd624b8a8d7aa8a348f112fe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35693 } } }
I20260926 03:37:21.761642 23393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3905cb4d9bf24e3593154f6c13c86855 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:37:21.762705 23297 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3905cb4d9bf24e3593154f6c13c86855 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 03:37:21.773762 23404 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3905cb4d9bf24e3593154f6c13c86855: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:37:21.773885 23404 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:37:21.770972 23230 master_runner.cc:186] Error getting master registration for 127.0.97.62:35411: OK, instance_id { permanent_uuid: "c744ec68c68a481eac6ca1223d9b1701" instance_seqno: 1790393840408529 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 03:37:21.783097 23230 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 03:37:21.816525 23326 tablet.cc:2369] T 00000000000000000000000000000000 P c48aac0cfd624b8a8d7aa8a348f112fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:37:21.844679 23223 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35693 } attempt: 5
Sep 26 03:37:21 dist-test-slave-dt7p krb5kdc[23143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790393840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 03:37:21.865772 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.845717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35693 (local address 127.0.0.1:50472)
0926 03:37:21.846102 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0926 03:37:21.846126 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:37:21.846286 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:37:21.846803 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:37:21.846810 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:37:21.847449 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:37:21.847711 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:21.847728 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.849245 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.849251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:21.850762 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:37:21.850780 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.851830 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.851836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:37:21.852037 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:21.852914 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:37:21.852942 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:37:21.856187 (+ 3245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:37:21.861008 (+ 4821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:21.861015 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:21.861020 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:21.861603 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:21.862207 (+ 604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:37:21.862213 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:37:21.862217 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:37:21.862506 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:37:21.863389 (+ 883us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:37:21.863394 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:37:21.863646 (+ 252us) client_negotiation.cc:769] Sending connection context
0926 03:37:21.863825 (+ 179us) client_negotiation.cc:240] Negotiation successful
0926 03:37:21.864519 (+ 694us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215}
I20260926 03:37:21.866344 23375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.845847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50472 (local address 127.0.97.60:35693)
0926 03:37:21.846095 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:21.846106 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:21.846270 (+ 164us) server_negotiation.cc:407] Connection header received
0926 03:37:21.846482 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:21.846492 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:21.846625 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:21.846756 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:21.847868 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.849111 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.850958 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.851680 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.851826 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:21.856446 (+ 4620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:21.856454 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:21.856459 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:21.856522 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:21.860826 (+ 4304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:21.861760 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:21.861767 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:21.861771 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:21.862065 (+ 294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:21.862652 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:21.862656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:21.862659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:21.863167 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:21.863321 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:21.864364 (+ 1043us) server_negotiation.cc:299] Negotiation successful
0926 03:37:21.865382 (+ 1018us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 03:37:21.874248 23223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c744ec68c68a481eac6ca1223d9b1701: Bootstrap starting.
I20260926 03:37:21.879418 23223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c744ec68c68a481eac6ca1223d9b1701: Neither blocks nor log segments found. Creating new log.
I20260926 03:37:21.881032 23223 log.cc:824] T 00000000000000000000000000000000 P c744ec68c68a481eac6ca1223d9b1701: Log is configured to *not* fsync() on all Append() calls
I20260926 03:37:21.886502 23223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c744ec68c68a481eac6ca1223d9b1701: No bootstrap required, opened a new log
I20260926 03:37:21.895103 23223 raft_consensus.cc:348] T 00000000000000000000000000000000 P c744ec68c68a481eac6ca1223d9b1701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744ec68c68a481eac6ca1223d9b1701" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35411 } } peers { permanent_uuid: "3905cb4d9bf24e3593154f6c13c86855" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45779 } } peers { permanent_uuid: "c48aac0cfd624b8a8d7aa8a348f112fe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35693 } }
I20260926 03:37:21.895576 23223 raft_consensus.cc:729] T 00000000000000000000000000000000 P c744ec68c68a481eac6ca1223d9b1701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c744ec68c68a481eac6ca1223d9b1701, State: Initialized, Role: FOLLOWER
I20260926 03:37:21.896157 23223 consensus_queue.cc:260] T 00000000000000000000000000000000 P c744ec68c68a481eac6ca1223d9b1701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744ec68c68a481eac6ca1223d9b1701" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35411 } } peers { permanent_uuid: "3905cb4d9bf24e3593154f6c13c86855" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45779 } } peers { permanent_uuid: "c48aac0cfd624b8a8d7aa8a348f112fe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35693 } }
I20260926 03:37:21.903739 23407 sys_catalog.cc:455] T 00000000000000000000000000000000 P c744ec68c68a481eac6ca1223d9b1701 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744ec68c68a481eac6ca1223d9b1701" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35411 } } peers { permanent_uuid: "3905cb4d9bf24e3593154f6c13c86855" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45779 } } peers { permanent_uuid: "c48aac0cfd624b8a8d7aa8a348f112fe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35693 } } }
I20260926 03:37:21.904156 23407 sys_catalog.cc:458] T 00000000000000000000000000000000 P c744ec68c68a481eac6ca1223d9b1701 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:37:21.906387 23223 sys_catalog.cc:564] T 00000000000000000000000000000000 P c744ec68c68a481eac6ca1223d9b1701 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:37:21.908514 23198 raft_consensus.cc:482] T 00000000000000000000000000000000 P c744ec68c68a481eac6ca1223d9b1701 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 03:37:21.908816 23198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c744ec68c68a481eac6ca1223d9b1701 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:21.913118 23198 raft_consensus.cc:504] T 00000000000000000000000000000000 P c744ec68c68a481eac6ca1223d9b1701 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744ec68c68a481eac6ca1223d9b1701" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35411 } } peers { permanent_uuid: "3905cb4d9bf24e3593154f6c13c86855" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45779 } } peers { permanent_uuid: "c48aac0cfd624b8a8d7aa8a348f112fe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35693 } }
I20260926 03:37:21.916478 23272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c744ec68c68a481eac6ca1223d9b1701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3905cb4d9bf24e3593154f6c13c86855"
I20260926 03:37:21.917102 23272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3905cb4d9bf24e3593154f6c13c86855 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:21.921200 23272 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3905cb4d9bf24e3593154f6c13c86855 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c744ec68c68a481eac6ca1223d9b1701 in term 1.
I20260926 03:37:21.922030 23175 leader_election.cc:302] T 00000000000000000000000000000000 P c744ec68c68a481eac6ca1223d9b1701 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3905cb4d9bf24e3593154f6c13c86855, c744ec68c68a481eac6ca1223d9b1701; no voters:
I20260926 03:37:21.922571 23407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c744ec68c68a481eac6ca1223d9b1701 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:37:21.930045 23347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c744ec68c68a481eac6ca1223d9b1701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c48aac0cfd624b8a8d7aa8a348f112fe"
I20260926 03:37:21.930663 23347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c48aac0cfd624b8a8d7aa8a348f112fe [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:37:21.934749 23347 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c48aac0cfd624b8a8d7aa8a348f112fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c744ec68c68a481eac6ca1223d9b1701 in term 1.
I20260926 03:37:21.935601 23407 raft_consensus.cc:686] T 00000000000000000000000000000000 P c744ec68c68a481eac6ca1223d9b1701 [term 1 LEADER]: Becoming Leader. State: Replica: c744ec68c68a481eac6ca1223d9b1701, State: Running, Role: LEADER
I20260926 03:37:21.936365 23407 consensus_queue.cc:237] T 00000000000000000000000000000000 P c744ec68c68a481eac6ca1223d9b1701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744ec68c68a481eac6ca1223d9b1701" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35411 } } peers { permanent_uuid: "3905cb4d9bf24e3593154f6c13c86855" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45779 } } peers { permanent_uuid: "c48aac0cfd624b8a8d7aa8a348f112fe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35693 } }
I20260926 03:37:21.939509 23198 leader_election.cc:288] T 00000000000000000000000000000000 P c744ec68c68a481eac6ca1223d9b1701 [CANDIDATE]: Term 1 election: Requested vote from peers 3905cb4d9bf24e3593154f6c13c86855 (127.0.97.61:45779), c48aac0cfd624b8a8d7aa8a348f112fe (127.0.97.60:35693)
I20260926 03:37:21.954830 23411 sys_catalog.cc:455] T 00000000000000000000000000000000 P c744ec68c68a481eac6ca1223d9b1701 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c744ec68c68a481eac6ca1223d9b1701. Latest consensus state: current_term: 1 leader_uuid: "c744ec68c68a481eac6ca1223d9b1701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744ec68c68a481eac6ca1223d9b1701" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35411 } } peers { permanent_uuid: "3905cb4d9bf24e3593154f6c13c86855" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45779 } } peers { permanent_uuid: "c48aac0cfd624b8a8d7aa8a348f112fe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35693 } } }
I20260926 03:37:21.955513 23411 sys_catalog.cc:458] T 00000000000000000000000000000000 P c744ec68c68a481eac6ca1223d9b1701 [sys.catalog]: This master's current role is: LEADER
W20260926 03:37:21.968909 23251 tablet.cc:2369] T 00000000000000000000000000000000 P 3905cb4d9bf24e3593154f6c13c86855: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:37:21.973562 23420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:37:22.008772 23420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:37:22.018488 23376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.982849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38606 (local address 127.0.97.62:35411)
0926 03:37:21.983059 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:21.983067 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:21.984151 (+ 1084us) server_negotiation.cc:407] Connection header received
0926 03:37:21.984466 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:21.984474 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:21.984607 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:21.984698 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:21.989868 (+ 5170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.991480 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.996306 (+ 4826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:21.997318 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:21.997463 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:22.007015 (+ 9552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:22.007024 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:22.007030 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:22.007110 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:22.013127 (+ 6017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:22.014315 (+ 1188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:22.014325 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:22.014332 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:22.014557 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:22.015051 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:22.015056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:22.015061 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:22.015549 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:22.015951 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:22.017360 (+ 1409us) server_negotiation.cc:299] Negotiation successful
0926 03:37:22.017998 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 03:37:22.035183 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.984037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36974 (local address 127.0.97.61:45779)
0926 03:37:21.984327 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:21.984333 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:21.999247 (+ 14914us) server_negotiation.cc:407] Connection header received
0926 03:37:22.000578 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:22.000587 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:22.000715 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:22.000811 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:22.003539 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:22.004801 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:22.008156 (+ 3355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:22.008950 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:22.009057 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:22.024965 (+ 15908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:22.024977 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:22.024989 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:22.025070 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:22.030431 (+ 5361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:22.031567 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:22.031577 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:22.031589 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:22.031825 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:22.032393 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:22.032402 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:22.032408 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:22.033012 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:22.033371 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:22.034181 (+ 810us) server_negotiation.cc:299] Negotiation successful
0926 03:37:22.034581 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260926 03:37:22.043434 23375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:21.986696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50478 (local address 127.0.97.60:35693)
0926 03:37:21.986930 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:21.986939 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:22.000059 (+ 13120us) server_negotiation.cc:407] Connection header received
0926 03:37:22.000237 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:22.000246 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:22.000371 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:22.000490 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:22.005059 (+ 4569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:22.006206 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:22.015914 (+ 9708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:22.016679 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:22.016800 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:22.022765 (+ 5965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:37:22.022776 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:37:22.022783 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:37:22.022840 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:37:22.038356 (+ 15516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:22.039172 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:22.039182 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:22.039188 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:22.039414 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:37:22.040027 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:37:22.040034 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:37:22.040040 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:37:22.040554 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:37:22.042284 (+ 1730us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:22.042459 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 03:37:22.042836 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 03:37:22.062965 23347 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c48aac0cfd624b8a8d7aa8a348f112fe [term 1 FOLLOWER]: Refusing update from remote peer c744ec68c68a481eac6ca1223d9b1701: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:37:22.065039 23407 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c744ec68c68a481eac6ca1223d9b1701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c48aac0cfd624b8a8d7aa8a348f112fe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:37:22.070133 23272 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3905cb4d9bf24e3593154f6c13c86855 [term 1 FOLLOWER]: Refusing update from remote peer c744ec68c68a481eac6ca1223d9b1701: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:37:22.071219 23407 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c744ec68c68a481eac6ca1223d9b1701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3905cb4d9bf24e3593154f6c13c86855" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:37:22.103971 23379 sys_catalog.cc:455] T 00000000000000000000000000000000 P c48aac0cfd624b8a8d7aa8a348f112fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader c744ec68c68a481eac6ca1223d9b1701. Latest consensus state: current_term: 1 leader_uuid: "c744ec68c68a481eac6ca1223d9b1701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744ec68c68a481eac6ca1223d9b1701" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35411 } } peers { permanent_uuid: "3905cb4d9bf24e3593154f6c13c86855" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45779 } } peers { permanent_uuid: "c48aac0cfd624b8a8d7aa8a348f112fe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35693 } } }
I20260926 03:37:22.104363 23379 sys_catalog.cc:458] T 00000000000000000000000000000000 P c48aac0cfd624b8a8d7aa8a348f112fe [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:37:22.105257 23430 mvcc.cc:205] Tried to move back new op lower bound from 7333453176974516224 to 7333453176578818048. Current Snapshot: MvccSnapshot[applied={T|T < 7333453176974516224}]
I20260926 03:37:22.115612 23379 sys_catalog.cc:455] T 00000000000000000000000000000000 P c48aac0cfd624b8a8d7aa8a348f112fe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c744ec68c68a481eac6ca1223d9b1701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744ec68c68a481eac6ca1223d9b1701" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35411 } } peers { permanent_uuid: "3905cb4d9bf24e3593154f6c13c86855" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45779 } } peers { permanent_uuid: "c48aac0cfd624b8a8d7aa8a348f112fe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35693 } } }
I20260926 03:37:22.115877 23379 sys_catalog.cc:458] T 00000000000000000000000000000000 P c48aac0cfd624b8a8d7aa8a348f112fe [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:37:22.116961 23393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3905cb4d9bf24e3593154f6c13c86855 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c744ec68c68a481eac6ca1223d9b1701. Latest consensus state: current_term: 1 leader_uuid: "c744ec68c68a481eac6ca1223d9b1701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744ec68c68a481eac6ca1223d9b1701" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35411 } } peers { permanent_uuid: "3905cb4d9bf24e3593154f6c13c86855" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45779 } } peers { permanent_uuid: "c48aac0cfd624b8a8d7aa8a348f112fe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35693 } } }
I20260926 03:37:22.117354 23393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3905cb4d9bf24e3593154f6c13c86855 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:37:22.118256 23432 mvcc.cc:205] Tried to move back new op lower bound from 7333453176974516224 to 7333453176578818048. Current Snapshot: MvccSnapshot[applied={T|T < 7333453176974516224}]
I20260926 03:37:22.121948 23411 sys_catalog.cc:455] T 00000000000000000000000000000000 P c744ec68c68a481eac6ca1223d9b1701 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c744ec68c68a481eac6ca1223d9b1701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744ec68c68a481eac6ca1223d9b1701" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35411 } } peers { permanent_uuid: "3905cb4d9bf24e3593154f6c13c86855" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45779 } } peers { permanent_uuid: "c48aac0cfd624b8a8d7aa8a348f112fe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35693 } } }
I20260926 03:37:22.122232 23411 sys_catalog.cc:458] T 00000000000000000000000000000000 P c744ec68c68a481eac6ca1223d9b1701 [sys.catalog]: This master's current role is: LEADER
I20260926 03:37:22.122524 23411 sys_catalog.cc:455] T 00000000000000000000000000000000 P c744ec68c68a481eac6ca1223d9b1701 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c744ec68c68a481eac6ca1223d9b1701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744ec68c68a481eac6ca1223d9b1701" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35411 } } peers { permanent_uuid: "3905cb4d9bf24e3593154f6c13c86855" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45779 } } peers { permanent_uuid: "c48aac0cfd624b8a8d7aa8a348f112fe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35693 } } }
I20260926 03:37:22.122730 23411 sys_catalog.cc:458] T 00000000000000000000000000000000 P c744ec68c68a481eac6ca1223d9b1701 [sys.catalog]: This master's current role is: LEADER
I20260926 03:37:22.123699 23420 catalog_manager.cc:1347] Generated new cluster ID: be45ae6bf1a8485ca1fbc2503672b322
I20260926 03:37:22.123842 23420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:37:22.133076 23393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3905cb4d9bf24e3593154f6c13c86855 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c744ec68c68a481eac6ca1223d9b1701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744ec68c68a481eac6ca1223d9b1701" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35411 } } peers { permanent_uuid: "3905cb4d9bf24e3593154f6c13c86855" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45779 } } peers { permanent_uuid: "c48aac0cfd624b8a8d7aa8a348f112fe" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35693 } } }
I20260926 03:37:22.133369 23393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3905cb4d9bf24e3593154f6c13c86855 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:37:22.161816 23420 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:37:22.166092 23420 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:37:22.194180 23420 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c744ec68c68a481eac6ca1223d9b1701: Generated new TSK 0
I20260926 03:37:22.195638 23420 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 03:37:22.335697 23376 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:37:22.325776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40612 (local address 127.0.97.62:35411)
0926 03:37:22.326032 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:22.326039 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:22.331990 (+ 5951us) server_negotiation.cc:407] Connection header received
0926 03:37:22.333791 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:22.333801 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:22.333924 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:22.334026 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:22.335363 (+ 1337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40612: BlockingRecv error: recv got EOF from 127.0.0.1:40612 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260926 03:37:22.338054 23375 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:37:22.322108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39766 (local address 127.0.97.60:35693)
0926 03:37:22.322368 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:22.322376 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:22.331505 (+ 9129us) server_negotiation.cc:407] Connection header received
0926 03:37:22.336348 (+ 4843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:22.336355 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:22.336458 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:22.336538 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:22.337776 (+ 1238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39766: BlockingRecv error: recv got EOF from 127.0.0.1:39766 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260926 03:37:22.340958 23302 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:37:22.332652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43416 (local address 127.0.97.61:45779)
0926 03:37:22.332873 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:22.332880 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:22.339152 (+ 6272us) server_negotiation.cc:407] Connection header received
0926 03:37:22.339501 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:22.339509 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:22.339621 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:22.339705 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:22.340693 (+ 988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43416: BlockingRecv error: recv got EOF from 127.0.0.1:43416 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260926 03:37:22.379495 23302 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:37:22.372949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43430 (local address 127.0.97.61:45779)
0926 03:37:22.373172 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:22.373180 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:22.377188 (+ 4008us) server_negotiation.cc:407] Connection header received
0926 03:37:22.377594 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:22.377602 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:22.377784 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:22.377883 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:22.379159 (+ 1276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43430: BlockingRecv error: recv got EOF from 127.0.0.1:43430 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260926 03:37:22.387104 23302 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:37:22.385318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43446 (local address 127.0.97.61:45779)
0926 03:37:22.385504 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:22.385511 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:22.385728 (+ 217us) server_negotiation.cc:407] Connection header received
0926 03:37:22.385893 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:22.385898 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:22.386015 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:22.386124 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:22.386845 (+ 721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43446: BlockingRecv error: recv got EOF from 127.0.0.1:43446 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260926 03:37:22.394172 23302 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:37:22.392125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43462 (local address 127.0.97.61:45779)
0926 03:37:22.392293 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:22.392299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:22.392775 (+ 476us) server_negotiation.cc:407] Connection header received
0926 03:37:22.392949 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:22.392954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:22.393031 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:22.393117 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:22.393942 (+ 825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43462: BlockingRecv error: recv got EOF from 127.0.0.1:43462 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 03:37:22.403554 23302 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:37:22.401700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43466 (local address 127.0.97.61:45779)
0926 03:37:22.401923 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:22.401947 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:22.402273 (+ 326us) server_negotiation.cc:407] Connection header received
0926 03:37:22.402430 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:22.402436 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:22.402523 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:22.402639 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:22.403332 (+ 693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43466: BlockingRecv error: recv got EOF from 127.0.0.1:43466 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
W20260926 03:37:22.417156 23302 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:37:22.414946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43468 (local address 127.0.97.61:45779)
0926 03:37:22.415201 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:22.415210 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:22.415725 (+ 515us) server_negotiation.cc:407] Connection header received
0926 03:37:22.415953 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:22.415958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:22.416029 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:22.416140 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:22.416918 (+ 778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43468: BlockingRecv error: recv got EOF from 127.0.0.1:43468 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
W20260926 03:37:22.444326 23302 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:37:22.442144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43478 (local address 127.0.97.61:45779)
0926 03:37:22.442394 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:22.442401 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:22.442752 (+ 351us) server_negotiation.cc:407] Connection header received
0926 03:37:22.442971 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:22.442977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:22.443098 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:22.443233 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:22.444060 (+ 827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43478: BlockingRecv error: recv got EOF from 127.0.0.1:43478 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
W20260926 03:37:22.482183 23302 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:37:22.479959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43488 (local address 127.0.97.61:45779)
0926 03:37:22.480188 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:22.480196 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:22.480602 (+ 406us) server_negotiation.cc:407] Connection header received
0926 03:37:22.480823 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:22.480829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:22.480956 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:22.481095 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:22.481890 (+ 795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43488: BlockingRecv error: recv got EOF from 127.0.0.1:43488 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260926 03:37:22.552162 23302 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:37:22.549924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43504 (local address 127.0.97.61:45779)
0926 03:37:22.550277 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:22.550285 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:22.550576 (+ 291us) server_negotiation.cc:407] Connection header received
0926 03:37:22.550775 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:22.550781 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:22.550903 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:22.551047 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:22.551875 (+ 828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43504: BlockingRecv error: recv got EOF from 127.0.0.1:43504 (error 108)
Metrics: {"server-negotiator.queue_time_us":150}
W20260926 03:37:22.686671 23302 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:37:22.684453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43514 (local address 127.0.97.61:45779)
0926 03:37:22.684739 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:22.684747 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:22.685188 (+ 441us) server_negotiation.cc:407] Connection header received
0926 03:37:22.685379 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:22.685384 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:22.685472 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:22.685579 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:37:22.686402 (+ 823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43514: BlockingRecv error: recv got EOF from 127.0.0.1:43514 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
I20260926 03:37:22.698664 23390 catalog_manager.cc:1259] Loaded cluster ID: be45ae6bf1a8485ca1fbc2503672b322
I20260926 03:37:22.698770 23390 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c48aac0cfd624b8a8d7aa8a348f112fe: loading cluster ID for follower catalog manager: success
I20260926 03:37:22.701632 23390 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c48aac0cfd624b8a8d7aa8a348f112fe: acquiring CA information for follower catalog manager: success
I20260926 03:37:22.703619 23390 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c48aac0cfd624b8a8d7aa8a348f112fe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 03:37:22.776340 23404 catalog_manager.cc:1259] Loaded cluster ID: be45ae6bf1a8485ca1fbc2503672b322
I20260926 03:37:22.776463 23404 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3905cb4d9bf24e3593154f6c13c86855: loading cluster ID for follower catalog manager: success
I20260926 03:37:22.779662 23404 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3905cb4d9bf24e3593154f6c13c86855: acquiring CA information for follower catalog manager: success
I20260926 03:37:22.781701 23404 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3905cb4d9bf24e3593154f6c13c86855: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 03:37:22.953824 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:22.947550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43522 (local address 127.0.97.61:45779)
0926 03:37:22.947797 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:22.947808 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:22.947980 (+ 172us) server_negotiation.cc:407] Connection header received
0926 03:37:22.948203 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:22.948208 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:22.948423 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:22.948535 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:37:22.949020 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:22.949797 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:22.951205 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:22.951646 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:22.951755 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:22.952342 (+ 587us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:37:22.953103 (+ 761us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:37:22.953251 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:22.953299 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 03:37:22.953381 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 03:37:22.967003 23502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:22.960195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39772 (local address 127.0.97.60:35693)
0926 03:37:22.960805 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:22.960810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:22.960829 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:37:22.961021 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:22.961025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:22.961216 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:22.961378 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:37:22.961861 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:22.962577 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:22.964445 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:22.964952 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:22.965057 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:22.965530 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:37:22.966181 (+ 651us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:37:22.966348 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:22.966484 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 03:37:22.966599 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":325,"threads_started":1}
I20260926 03:37:22.980880 23502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:22.973496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39780 (local address 127.0.97.60:35693)
0926 03:37:22.974843 (+ 1347us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:22.974847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:22.974862 (+ 15us) server_negotiation.cc:407] Connection header received
0926 03:37:22.974916 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:22.974920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:22.974990 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:22.975093 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:37:22.975661 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:22.976787 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:22.979033 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:22.979455 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:22.979590 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:22.980032 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:37:22.980141 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:37:22.980242 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:22.980426 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 03:37:22.980520 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1231}
I20260926 03:37:22.982577 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:22.974197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43528 (local address 127.0.97.61:45779)
0926 03:37:22.974445 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:22.974454 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:22.976017 (+ 1563us) server_negotiation.cc:407] Connection header received
0926 03:37:22.976217 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:22.976224 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:22.976331 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:22.976431 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:37:22.977361 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:22.978150 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:22.980368 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:22.981079 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:22.981230 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:22.981727 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:37:22.981884 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:37:22.981952 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:22.982192 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 03:37:22.982291 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 03:37:22.983971 23508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:37:22.973341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40626 (local address 127.0.97.62:35411)
0926 03:37:22.975614 (+ 2273us) server_negotiation.cc:206] Beginning negotiation
0926 03:37:22.975623 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:37:22.975642 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:37:22.975807 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:37:22.975812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:37:22.976078 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:37:22.976293 (+ 215us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:37:22.977398 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:22.978567 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:22.981350 (+ 2783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:37:22.981831 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:37:22.981981 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:37:22.982965 (+ 984us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:37:22.983374 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:37:22.983503 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 03:37:22.983601 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 03:37:22.983702 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2102,"thread_start_us":1973,"threads_started":1}
I20260926 03:37:22.989264 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 23156
I20260926 03:37:22.990483 23218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:37:23.121759 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 23156
I20260926 03:37:23.147962 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 23230
I20260926 03:37:23.149464 23292 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:37:23.275622 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 23230
I20260926 03:37:23.300401 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskssCcLE/build/asan/bin/kudu pid 23305
I20260926 03:37:23.301805 23367 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:37:23.423656 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskssCcLE/build/asan/bin/kudu with pid 23305
2026-09-26T03:37:23Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4998 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10037 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (380162 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST